One of the great things about EVE Online is that a very large portion of the people who play it work in the field of IT. This results in a huge amount of third party tools (EVEMon, EVE Mentat, Elinor, EVE-Central, most alliance's websites, and the list goes on and on...) as well as a great deal of support for developer's from CCP's end. The possibilities are nigh endless in terms of the applications that can be developed with some code know-how, some hard work, plenty of Googling, and a vision.
It all starts with an idea. As most of my readers will know, one of my favorite/main activities in EVE is station trading, which is where I make much of my ISK (and try to put in as little effort as possible). However, when it comes to the things I've tried coding to make station trading more profitable, I've put in many hours of effort, but I dare say that sometimes, I've enjoyed them more than EVE (looking at you, siege fleets). One of the most difficult parts of station trading is deciding which items you'll be trading to turn a profit. A lot of factors must be considered when making the decision, such as the stability of the item, the volume traded, the profit margin, the cost, etc. The end result is that although a profitable item is not hard to find, one that will give consistent and high returns can be. Then it hit me: why don't I make a program that will tell me what items offer the highest margins and meet my criteria for items that I want to trade?
I'm no code genius, but I have limited experience in web design (HTML, CSS, PHP, and Javascript/jQuery), as well as software development (Python and some C#). My original plan was to design my program in a way that allowed the entire thing to be operated from the IGB in EVE. The program would first scan the market (every item) to put them in the cache, then scrape the cache, calculate the margin for each item, filter the items through some criteria, and finally output the results to a web page that would be readable in the IGB. My hopes were dashed as I quickly found that such a system would be much more trouble than it is worth.
To begin with, I needed a webpage that I could load in the IGB that would scan the market to load every item into the cache. Rather than start from scratch, I butchered my code from the order scanner page of masterpiece of EVE market software known as EVE Mentat. Using that as a starting point, I now had an HTML page that would open the market details for every item in my list. To make life easier, I set up a simple HTTP server with this script (courtesy of stackoverflow). Simply save that as a .py file in the same directory as your HTML file (which you should name index.html) and run it. You can access the file by going to 127.0.0.1:8000 in the IGB.
Now I found myself with a webpage capable of scanning the market, but no item IDs for which to scan. I downloaded a data dump (which was from Incarna, so I'm probably missing some items but they shouldn't be too big of a deal) from the EVE forums here and through the magic of MySQL pulled out a list of every item in the database which exists on the market (some items can't be purchased on the market, but a quick Google search taught me how to select only those that can be). This data was placed in the HTML page I had made earlier and when I ran it, it worked! However, the import turned up over 6000 items, meaning that with a 3 second interval (the minimum I'd need to use to avoid going over the maximum number of market requests I could make, which would make the script continue to run without actually pulling more data), it would take hours upon hours to scan the market. I solved this by just letting it run while I slept.
At this stage, I was left with the market data for (almost) every item on the market in my cache. What to do from here? I used the dumper tool from EVE-Central to scrape the cache and export the market orders for every item into one CSV file. This process unsurprisingly took quite a while. However, it also meant that much of the hard part was done, as Python was undoubtedly my strongest language of all those I had used so far, and all that remained was to use it to process the data into a user-friendly format. Some quick code calculations yielded a script which analyzed the orders for every item, threw out those with no data or those with only one type (buy or sell) of order, calculated the % profit margin for each item (taking my skills into account), and threw away any item with less than 30 mil worth of potential profit currently on the market. Entity's Reverence library provided me with the means to pull the names for the items from their item IDs. The results were exported to another HTML file in the form of a table, which using some jQuery magic, gave me the ability to sort the data by column. I included columns for volume and margin, so I could play with the data as I wished. I also made each item's name into a link that would open the market data for that item, to make checking on/trading desirable items easier.
I am fairly happy with the end result, and it's not hard to see why; I now have an easy to use table that contains data on most of the items worth station trading. I only actually went through the list for a few minutes, but in those few minutes I found at least two new items to invest in with profit margins in the thousands of percents that looked very promising.
For the time being, I do not think I will be releasing this file. Though I love helping the community, which gives me indirect competition, I have never released any of the items I am trading in because I would like to avoid directly competing with my readers as much as possible. Releasing this file to the masses would make my life absolute hell whenever I try to station trade. However, I spent about two days making this program, and I did not have available to me (most of) a step by step guide on how to do so. Anyone with a simple background in code or the ability to Google should be able to recreate a lot of what I've done here simply based on reading the above guide. Though I probably shouldn't, I have a soft spot for helping people out, so if you have any questions, leave a comment or PM me on reddit (/u/toxicity959) and I will see if I can help you work through whatever problems you may have encountered.
Thanks for reading my blog and keep on developing! Fly safe o7
It all starts with an idea. As most of my readers will know, one of my favorite/main activities in EVE is station trading, which is where I make much of my ISK (and try to put in as little effort as possible). However, when it comes to the things I've tried coding to make station trading more profitable, I've put in many hours of effort, but I dare say that sometimes, I've enjoyed them more than EVE (looking at you, siege fleets). One of the most difficult parts of station trading is deciding which items you'll be trading to turn a profit. A lot of factors must be considered when making the decision, such as the stability of the item, the volume traded, the profit margin, the cost, etc. The end result is that although a profitable item is not hard to find, one that will give consistent and high returns can be. Then it hit me: why don't I make a program that will tell me what items offer the highest margins and meet my criteria for items that I want to trade?
I'm no code genius, but I have limited experience in web design (HTML, CSS, PHP, and Javascript/jQuery), as well as software development (Python and some C#). My original plan was to design my program in a way that allowed the entire thing to be operated from the IGB in EVE. The program would first scan the market (every item) to put them in the cache, then scrape the cache, calculate the margin for each item, filter the items through some criteria, and finally output the results to a web page that would be readable in the IGB. My hopes were dashed as I quickly found that such a system would be much more trouble than it is worth.
To begin with, I needed a webpage that I could load in the IGB that would scan the market to load every item into the cache. Rather than start from scratch, I butchered my code from the order scanner page of masterpiece of EVE market software known as EVE Mentat. Using that as a starting point, I now had an HTML page that would open the market details for every item in my list. To make life easier, I set up a simple HTTP server with this script (courtesy of stackoverflow). Simply save that as a .py file in the same directory as your HTML file (which you should name index.html) and run it. You can access the file by going to 127.0.0.1:8000 in the IGB.
Now I found myself with a webpage capable of scanning the market, but no item IDs for which to scan. I downloaded a data dump (which was from Incarna, so I'm probably missing some items but they shouldn't be too big of a deal) from the EVE forums here and through the magic of MySQL pulled out a list of every item in the database which exists on the market (some items can't be purchased on the market, but a quick Google search taught me how to select only those that can be). This data was placed in the HTML page I had made earlier and when I ran it, it worked! However, the import turned up over 6000 items, meaning that with a 3 second interval (the minimum I'd need to use to avoid going over the maximum number of market requests I could make, which would make the script continue to run without actually pulling more data), it would take hours upon hours to scan the market. I solved this by just letting it run while I slept.
At this stage, I was left with the market data for (almost) every item on the market in my cache. What to do from here? I used the dumper tool from EVE-Central to scrape the cache and export the market orders for every item into one CSV file. This process unsurprisingly took quite a while. However, it also meant that much of the hard part was done, as Python was undoubtedly my strongest language of all those I had used so far, and all that remained was to use it to process the data into a user-friendly format. Some quick code calculations yielded a script which analyzed the orders for every item, threw out those with no data or those with only one type (buy or sell) of order, calculated the % profit margin for each item (taking my skills into account), and threw away any item with less than 30 mil worth of potential profit currently on the market. Entity's Reverence library provided me with the means to pull the names for the items from their item IDs. The results were exported to another HTML file in the form of a table, which using some jQuery magic, gave me the ability to sort the data by column. I included columns for volume and margin, so I could play with the data as I wished. I also made each item's name into a link that would open the market data for that item, to make checking on/trading desirable items easier.
I am fairly happy with the end result, and it's not hard to see why; I now have an easy to use table that contains data on most of the items worth station trading. I only actually went through the list for a few minutes, but in those few minutes I found at least two new items to invest in with profit margins in the thousands of percents that looked very promising.
For the time being, I do not think I will be releasing this file. Though I love helping the community, which gives me indirect competition, I have never released any of the items I am trading in because I would like to avoid directly competing with my readers as much as possible. Releasing this file to the masses would make my life absolute hell whenever I try to station trade. However, I spent about two days making this program, and I did not have available to me (most of) a step by step guide on how to do so. Anyone with a simple background in code or the ability to Google should be able to recreate a lot of what I've done here simply based on reading the above guide. Though I probably shouldn't, I have a soft spot for helping people out, so if you have any questions, leave a comment or PM me on reddit (/u/toxicity959) and I will see if I can help you work through whatever problems you may have encountered.
Thanks for reading my blog and keep on developing! Fly safe o7
Have a look into EMDR, if you want to look at market data.
ReplyDeleteIf you want an up to data database conversion: https://www.fuzzwork.co.uk/dump/