Бюджет по договоренности
We are searching for experienced Java-developer
We are looking for experienced Java developer for 1 project job. Please write in propositions price and needed time to finish this things( +3 more tasks...symbols limit are here). Thanx, waiting for Your feedbacks.
Requirements for Java developer:
- at least 2 years of experience on the position of Java developer OR finished computer science studies + 1 year of experience OR finished computer science studies + engineer/bachelor/master degree work involving Java OR Java Sun Certificate
- some experience with Java SE development
- knowledge of mysql database and general db-related things (sql, db design, index and query optimization)
1. Easiest one. We need a process that will take data from import from Yahoo in one database and put it into other database, possibly on other server. Rather straightforward thing but should be bullet proof – if it fails (db exceptions, no connection etc.) it should restore from the place where it was without missing any data and should "get up running" without need to start it by hand. So preferably it should be a CRON job that will check once per 5 minutes if other instances are running, do some part of job (for example run for an hour), end itself, then another instance starts and run (if there is something to copy)***.
Also it needs to take only data that we want – selected set of markets.
Preferably Java SE with JDBC or JPA. Without UI.
Source db structure (has about 130 000 000 rows, will have more)
Target db structure:
CREATE TABLE `quotations` (
`ticker` varchar(50) NOT NULL,
`date` datetime NOT NULL,
`open` double DEFAULT NULL,
`high` double DEFAULT NULL,
`low` double DEFAULT NULL,
`close` double DEFAULT NULL,
`volume` double DEFAULT NULL,
PRIMARY KEY (`ticker`,`date`),
KEY `date_index` (`date`),
KEY `ticker_index` (`ticker`),
) ENGINE=InnoDB DEFAULT CHARSET=latin2
There is many to one relation between historical and tickers in source db. Symbol from tickers in source db should transpose into ticker in target db. Rest should be rather self-descriptory.
- check if those still work and import into database of similar structure as in task 1 (beginning from moment where data ends)
- if doesn't work then update them to work
- make them a bit more reliable – so check what happens when connection to db or yahoo is lost, make sure it will restart from a place where it was without any holes in data, make sure it makes it's job by chunks of work that last for at most 2h and after this time it should finish some part, end and then there should be started a new instance starting from place where last one was***.
3. We need to get current lists of tickers (symbols) from XTB, Saxo Bank Trader, Plus500, City Index trading platforms.
For XTB it shouldn't be hard – they allow to export it as far as I remember, just need to apply for test account, download Meta Trader platform and get it.
For Saxo Bank their web interface can be used with web developer console to make some JS to put through console that will copy symbols from the tree. Tree needs to be opened (it isn't fully preloaded, loaded by ajax), so it might either need to open each category in tree by click or write some script that will trigger opening of them. We did something like that nearly year ago but don't have script now (it was done with traversing by hand, it put symbols list into alert and it took ctrl+A ctrl+C ctrl+v to copy it from there).
For City Index their system is well secured so it will be hard to get them. Communication is through secure https (packet sniffer won't help) and it's secured against man in the middle approach (as far as Charles software won't help, maybe some better soft would), hard to decompile... We can get a list of markets by just asking (as we did once) but they give it without symbols, only full name. Someone would need to think a bit about getting it a – if it's possible to get from platform somehow. If couldn't find a way then we will skip on this one.
We didn't try to get them from Plus500 so it's new ground, will need to find out some way.
4. We need to try to get a probe of data for each symbol from each of those trading providers. It can be even a month back in history and can be for example only dayly close values. Where this data cannot be taken then we need to try to get one last value (preferably a close for some day).
How - one will need to be creative here.
5. We need to match symbols between those lists and Yahoo together accurately.
It can be done in few steps (or think about better solution):
- we match exact same symbols to get some pairs
- we match symbol names by some string metric (like Jaro-Winkler distance, Levenshtein distance, Hamming distance), need to find out which metric suits problem best and take only pairs that are above some level (which also should be chosen to clear out most of negatives but not allow much false negatives if we have history data or not allow much false positives if we can't have history data for those)
- for those that we have history data we need to match it with yahoo history (from the table same as in task 1) and compare, if difference is small in relation to price level (in means of variation of difference) then it's a good match
- we need to take those pairs that match well by symbol name plus match well on history data (or very well on symbol - 100% same or 1 character difference - and we don't have data to match)
Technology: preferably command line script/program or set of those written in Java SE 1.6 or Pentaho ETL or C++
We have some partial solution in Pentaho ETL here (I will search for it, might help).