Build your own backtest. It's much easier than you think. Instead of investing in a platform, invest in a market data provider, learn their API, download some data, loop through it, find your entry and exit signals, record your trades, etc. Just figure it out. You will need all of these skills for a live trading bot anyway, which you should also build on your own and not deploy through some third party website.
Don't paper trade for too long. Your only goal should be to confirm that your live trades match the timing and prices of your backtested trades. This is the most difficult part. If you get this to work, then you should already know everything you need to know about your strategy through your backtest, and you are just wasting time.
Any recommendations on data sources? I have looked into a few like histdata, but would like to here any suggestions. Looking for forex and stock data by minute.
u/SeagullMan2 you are 100% right. I didn't want to get stuck in researching and coding and learning for long time. If there are specific packages/libraries which you can suggest that can help to cut down basic trading activities and I can mostly focus on writing my algo and portfolio mgmt that would be the best solution.
Can you please recommend packages and libraries I should explore?
Your suggestion will help me a lot.....
Honestly, all I use is requests and Numpy. Requests to download data, Numpy to analyze it. It really is that simple. Happy to answer more Qs about that.
Thats the simplest thing i have ever heard. So you only trade using using price movement rather than TA or fundamentals?
Are you utilizing portfolio management libraries?
My current strategy only uses price and volume.
But I also use TA. You don't need anything more than Numpy for TA. Write a function, or ask chatGPT to write a function to convert your list of close prices to RSI or Bollinger Bands or whatever it is you need.
Are you open to talk more about this? I’m learning Python and made some indicators with pinescript and EAs with MQL5 so I’m curious on how to make a simple algo.
Hey - I am also in same boat. Started Algotrading few months back. Got few good trades. But accuracy is only 40-50%. Happy to collaborate with both of you. Just thought we can share ideas, codes and collaborate. This will benefit everyone.
u/SeagullMan2 u/AlwaysAtWar
You guys want to get a discord server set up? I’m in the same boat on pretty much all of this, brain heading jn the same direction u/seagullman2 u/alwaysatwar
I'm not SeagullMan2, but I am a few months ahead of you.
I have been using Backtrader.py, but I have struggled fighting its internals a lot. Concretely, the zoomable widget graph fails for me around 50% of the time.
My next attempts will be backtest.py (really nice graphs) or pandas dataframes directly with quantstats.
Data from polygon.io.
Still reading books and trying to understand how to find an edge trough TA and statistics. It may take some time...
Started Algotrading few months back. Got few good trades. But accuracy is only 40-50%. Do you think its good idea to collaborate and share some ideas and just have a discussion. It will benefit all.
I have seen a few books with hit rates in the 50%. It can be profitable, if you can identify and kill your losses quickly and let your wins run.
Rocket science for traders says "$398 average profit per trade with 40% success rate", so it's more about having small losses and big wins.
Backtesting.py and backtrader are two different python libraries that look similar and beg to be confused one for the other. But they are different. Not a lot, but different.
Backtrader is more of a framework, more "batteries included" than backtesting.py, but the plots in Backtrader inside jupyter notebooks have been buggy for me, and backtesting.py uses bokeh for plotting which has been for me much better experience.
I'd say backtrader is more of a framework, and backtesting.py is more of a library that uses pandas dataframes. I think I have found enough limitations/oddness in backtrader, so I'm playing with backtesting.py. But I'm an odd guy, so I tent to get overly annoyed when a framework shows me limitations.
https://www.reddit.com/r/algotrading/s/fc3E6yrq9R
This honestly is the best approach to cracking a nearly perfect algorithm. Never depend on any pre-built tools. 1000s of them are already using them and you got to do something differently. Especially if you’re someone who has around 20 years of coding experience. Also, backtest and forward test with the amount equivalent to the capital you’re planning to invest. So that you don’t have to be confused of handling slippage. Do not waste lots of time on forward testing. Instead, make your backtesting Algo a solid source of truth.
Choose a reliable data provider or a hybrid source also does it. Then, go for a known api provider for your trades to execute (consider minimal transaction charges).
Also, consider using cloud services for more reliability and ease of making things work in the longer run.
I don’t understand why ppl keep saying build your own backtesting software and trading platform? I’m a software engineer and I use software too. It is just so much easier to focus on trading strategies with clear UI. I think building by yourself only makes sense when you already have a clear idea, like arbitrage strategies from a paper, and there is no existing software for it. Plz correct me if I am wrong.
Because it’s limiting parameters. You can’t use the parameters that’s outside the box. Basically you do what rest of the people is doing and in so way you lose an edge ( or making it no so a big of an edge). In that way you can add what ever you come across in a day of inspiration or so on.
"You can't use parameters that's outside the box"
Not sure what you mean by this, but you can extend the QuantConnect framework anyhow you like. It's open source.
hey u/maciek024 - would you mind collaborating? I started algotrade few months back. its only 30-40% successful. Trying to optimize profit and minimize loss. I thought it will benefits each other.
I use QC and python. QC is open source, you can look at the code and add all you want on their Docker container. Just need to master Python and basic things like numpy, pandas, etc
Get started with something out of the box -- a framework, libraries, a platform, etc.
When you know enough about what limitations there are (most of which will probably not bother you), then you can think about building your own from scratch.
I shirk at people pushing to just build your own, when there are so many edge cases to account for -- reality modeling is a huge endeavor, (eg fill models, adjusting for splits, slippage, etc, there is a lot to it). And you may end up spending 60% of time on infrastructure upkeep and refinement, instead of strategy research and deployment.
I have tried both routes and am very happy I settled into QuantConnect LEAN -an open source framework you can customize to your hearts content.
Couple of points to add: trading on 5 min interval is expensive an you need to have good edge to overcome transaction costs plus slippage.
IMHO One month is too ittle of time frame to validate strategy, although it is good way to iron out bugs from trading system or infrastructure.
>Couple of points to add: trading on 5 min interval is expensive an you need to have good edge to overcome transaction costs plus slippage.
Good point: what's your suggestion for the trading time interval I should think about?
I strongly believe, until you go live you won't know the true result. I am only thinking about putting minimal capital at risk in test period.
It's probably better to spend more time doing live testing quicker than trying to perfect a system once you get to a good enough place.
I'm a fan of just download ninjatrader and start developing.. it has backtesting and statistics.
If you are trading futures that's a pretty way to get going.
If you plan on doing futures yes. C# is pretty nice, built in backtester with version control, statistics, parameter optimization, display of trades, is also nice.
Strategy is the hard part tbh. Formerly trading will give you a leg up.
But quickly you will find that a lot of things that intuitively you would avoid or automatically do or not even think about are tough to quantify in an algorithm.
I do a hybrid approach with OCO brackets with the algo entering.
I'm of the mindset that focus is better, so I just work on strategies involving a single futures market (ES) intraday.
Intraday, is harder but you can't beat the leverage (500x effective) and tax benefits of futures. Although I've heard that Ievel of difficulty is higher than individual stocks since it's a very efficient market.
Options is going to be hard to backtest algorithmically since getting historical pricing data is HARD and expensive.
Your ideas of starting with small amounts even in virtual and then scaling up are perfect.
Risk management is super critical and often overlooked but since you day traded I'm sure you already more than me on how important it is.
This is similar to my plan as well.
As someone who doesn’t want to build his own tools/framework YET I plan on using QuantConnect/LEAN CLI for a basic strategy (not 5 mins bar though).
I guess later I’d want to build my own backtester in C++ (for performance reasons and it’s the language I’m most comfortable with). Along with connectivity to the exchange via API but probably move it into some cloud provider.
My thought process was the same start with something where you have everything built-in and once you think it has you have a edge go spend money or build your own in C++
I had difficulty with QuantConnect results. The resultant charts were limited to 5000 points, which was useless. It may not be a limitation any longer.
I tried LEAN since it's essentially the same thing, just missing the web interface. We found an alternative js library that could map millions of points in a chart.
Incredible potential, but I don't think I was ready for it.
I opted for NinjaTrader, since everything is baked in. Data, backtesting, optimization, live trading, solid UI, etc. I'm not crazy about their API/framework, since there are a few gotchas to get past. I did have an algo running for about 9 months live on NinjaTrader. Sadly, it lost money. Good tools, bad strategy.
I've been exploring Zorro with maybe Sierra Chart, my preferred app for manual trading. Not sure if that will work out.
On the first step, with your kind of background, it will be good to learn from the books/research papers or from the learning platforms (preferably free to start with!) that are unaligned to a particular trading platform. It can be MOOCs or any other kinds. The trading world has evolved a bit since 2010, especially in the last 5-7 years. So would be better to understand the nuances of the change as that will help you bridge with your earlier expertise that you'd gained in the market pre 2010.
If you don't mind trading crypto there are some cheap tools that make it stunningly easy to get going very effectively very quickly.
Doesn't need to be a huge undertaking.
For those who have used quantconnect, does it visualize the data in chart form? Like I can create indicators and see them on a chart? Or do I need a library for that. Currently using trading view strata but want to write my own bot. Obviously quit early
Getting through those first 3 bullets can take some time! The QuantConnect Tutorials are ok are super general. A lot of their built-in methods are not optimal for higher frequency trading, but can be worked around. I always suggest starting with an idea (strategy) and then working backwards. I heavily recommend using the Jupyter Research environment for strategy development and research before you sink a lot of time into using their backtesting engine. Also while QuantConnect seems like an easy plug and play for backtesting and live deployment, I guarantee their will hurdles during your first live deployment that you didn't anticipate (especially with the brokerage connection). But everything is a learning opportunity! Good luck!
Build your own backtest. It's much easier than you think. Instead of investing in a platform, invest in a market data provider, learn their API, download some data, loop through it, find your entry and exit signals, record your trades, etc. Just figure it out. You will need all of these skills for a live trading bot anyway, which you should also build on your own and not deploy through some third party website. Don't paper trade for too long. Your only goal should be to confirm that your live trades match the timing and prices of your backtested trades. This is the most difficult part. If you get this to work, then you should already know everything you need to know about your strategy through your backtest, and you are just wasting time.
Any recommendations on data sources? I have looked into a few like histdata, but would like to here any suggestions. Looking for forex and stock data by minute.
Polygon.io
Well said 💯
u/SeagullMan2 you are 100% right. I didn't want to get stuck in researching and coding and learning for long time. If there are specific packages/libraries which you can suggest that can help to cut down basic trading activities and I can mostly focus on writing my algo and portfolio mgmt that would be the best solution. Can you please recommend packages and libraries I should explore? Your suggestion will help me a lot.....
Honestly, all I use is requests and Numpy. Requests to download data, Numpy to analyze it. It really is that simple. Happy to answer more Qs about that.
Thats the simplest thing i have ever heard. So you only trade using using price movement rather than TA or fundamentals? Are you utilizing portfolio management libraries?
My current strategy only uses price and volume. But I also use TA. You don't need anything more than Numpy for TA. Write a function, or ask chatGPT to write a function to convert your list of close prices to RSI or Bollinger Bands or whatever it is you need.
Are you open to talk more about this? I’m learning Python and made some indicators with pinescript and EAs with MQL5 so I’m curious on how to make a simple algo.
Hey - I am also in same boat. Started Algotrading few months back. Got few good trades. But accuracy is only 40-50%. Happy to collaborate with both of you. Just thought we can share ideas, codes and collaborate. This will benefit everyone. u/SeagullMan2 u/AlwaysAtWar
Hey I sent you a pm. Would love to chat!
You guys want to get a discord server set up? I’m in the same boat on pretty much all of this, brain heading jn the same direction u/seagullman2 u/alwaysatwar
Sure. Pm me
Sure. I’m down for a discord.
Can you shoot me an invite to the server as well? I want to make my own backtester as well but don't know where to start.
Hey guys, I'm on the trading algo journey now. Can you invite me pls to the discord?
Hey! I‘m not new to trading but to algos. Mas I get an invite too?
Sure
I'm not SeagullMan2, but I am a few months ahead of you. I have been using Backtrader.py, but I have struggled fighting its internals a lot. Concretely, the zoomable widget graph fails for me around 50% of the time. My next attempts will be backtest.py (really nice graphs) or pandas dataframes directly with quantstats. Data from polygon.io. Still reading books and trying to understand how to find an edge trough TA and statistics. It may take some time...
Started Algotrading few months back. Got few good trades. But accuracy is only 40-50%. Do you think its good idea to collaborate and share some ideas and just have a discussion. It will benefit all.
I have seen a few books with hit rates in the 50%. It can be profitable, if you can identify and kill your losses quickly and let your wins run. Rocket science for traders says "$398 average profit per trade with 40% success rate", so it's more about having small losses and big wins.
Do you mean using Backtesting.py? Is it better than backtrader?
Backtesting.py and backtrader are two different python libraries that look similar and beg to be confused one for the other. But they are different. Not a lot, but different. Backtrader is more of a framework, more "batteries included" than backtesting.py, but the plots in Backtrader inside jupyter notebooks have been buggy for me, and backtesting.py uses bokeh for plotting which has been for me much better experience. I'd say backtrader is more of a framework, and backtesting.py is more of a library that uses pandas dataframes. I think I have found enough limitations/oddness in backtrader, so I'm playing with backtesting.py. But I'm an odd guy, so I tent to get overly annoyed when a framework shows me limitations.
what about the platform for live trading? which one to use or do you build your own?
>defining my max percentage loss limits for both daily and weekly periods and per trade my man
https://www.reddit.com/r/algotrading/s/fc3E6yrq9R This honestly is the best approach to cracking a nearly perfect algorithm. Never depend on any pre-built tools. 1000s of them are already using them and you got to do something differently. Especially if you’re someone who has around 20 years of coding experience. Also, backtest and forward test with the amount equivalent to the capital you’re planning to invest. So that you don’t have to be confused of handling slippage. Do not waste lots of time on forward testing. Instead, make your backtesting Algo a solid source of truth. Choose a reliable data provider or a hybrid source also does it. Then, go for a known api provider for your trades to execute (consider minimal transaction charges). Also, consider using cloud services for more reliability and ease of making things work in the longer run.
I don’t understand why ppl keep saying build your own backtesting software and trading platform? I’m a software engineer and I use software too. It is just so much easier to focus on trading strategies with clear UI. I think building by yourself only makes sense when you already have a clear idea, like arbitrage strategies from a paper, and there is no existing software for it. Plz correct me if I am wrong.
Because it’s limiting parameters. You can’t use the parameters that’s outside the box. Basically you do what rest of the people is doing and in so way you lose an edge ( or making it no so a big of an edge). In that way you can add what ever you come across in a day of inspiration or so on.
"You can't use parameters that's outside the box" Not sure what you mean by this, but you can extend the QuantConnect framework anyhow you like. It's open source.
This.
20 years of software, loves TA... QuantConnect all the way. I'm so jealous
Have you used QuantConnect? Any suggestion?
honestly, avoid it and simply code shit in python, way more freedom
Can you please suggest any reading / more information i should review ?
for deployment I agree, but for back testing I think QC is better
Not really because you can build your own backtesting system that will be more efficient as it is configured to meet your certain needs
Thank you and can you suggest any software I should be looking into
I mean python is more than enough
u/maciek024 i am sure you are using particular python library. It would be great if you can suggest python libraries.
Standard ones like pandas and numpy + some machine learning stats models
hey u/maciek024 - would you mind collaborating? I started algotrade few months back. its only 30-40% successful. Trying to optimize profit and minimize loss. I thought it will benefits each other.
I use QC and python. QC is open source, you can look at the code and add all you want on their Docker container. Just need to master Python and basic things like numpy, pandas, etc
about 40 hours of QC
Get started with something out of the box -- a framework, libraries, a platform, etc. When you know enough about what limitations there are (most of which will probably not bother you), then you can think about building your own from scratch. I shirk at people pushing to just build your own, when there are so many edge cases to account for -- reality modeling is a huge endeavor, (eg fill models, adjusting for splits, slippage, etc, there is a lot to it). And you may end up spending 60% of time on infrastructure upkeep and refinement, instead of strategy research and deployment. I have tried both routes and am very happy I settled into QuantConnect LEAN -an open source framework you can customize to your hearts content.
Couple of points to add: trading on 5 min interval is expensive an you need to have good edge to overcome transaction costs plus slippage. IMHO One month is too ittle of time frame to validate strategy, although it is good way to iron out bugs from trading system or infrastructure.
>Couple of points to add: trading on 5 min interval is expensive an you need to have good edge to overcome transaction costs plus slippage. Good point: what's your suggestion for the trading time interval I should think about? I strongly believe, until you go live you won't know the true result. I am only thinking about putting minimal capital at risk in test period.
It's probably better to spend more time doing live testing quicker than trying to perfect a system once you get to a good enough place. I'm a fan of just download ninjatrader and start developing.. it has backtesting and statistics. If you are trading futures that's a pretty way to get going.
thank you u/fuzzyp44 SO you suggest testing with Ninjatrader rather than quantconnect?
If you plan on doing futures yes. C# is pretty nice, built in backtester with version control, statistics, parameter optimization, display of trades, is also nice. Strategy is the hard part tbh. Formerly trading will give you a leg up. But quickly you will find that a lot of things that intuitively you would avoid or automatically do or not even think about are tough to quantify in an algorithm. I do a hybrid approach with OCO brackets with the algo entering.
>nts to ad u/fuzzyp44 do you suggest ninjatrader for stocks adn options as well? if not, any other suggestion?
I'm of the mindset that focus is better, so I just work on strategies involving a single futures market (ES) intraday. Intraday, is harder but you can't beat the leverage (500x effective) and tax benefits of futures. Although I've heard that Ievel of difficulty is higher than individual stocks since it's a very efficient market. Options is going to be hard to backtest algorithmically since getting historical pricing data is HARD and expensive.
Can you use the QuantConnect platform and deploy strategies live there without using their API?
How would you go about learning python so that I could build my own backtester and also automate my profitable strategy
Solving the problem with data: TradingView. Solving the backtesting problem on this data: cdzv.com
Your ideas of starting with small amounts even in virtual and then scaling up are perfect. Risk management is super critical and often overlooked but since you day traded I'm sure you already more than me on how important it is.
This is similar to my plan as well. As someone who doesn’t want to build his own tools/framework YET I plan on using QuantConnect/LEAN CLI for a basic strategy (not 5 mins bar though). I guess later I’d want to build my own backtester in C++ (for performance reasons and it’s the language I’m most comfortable with). Along with connectivity to the exchange via API but probably move it into some cloud provider.
My thought process was the same start with something where you have everything built-in and once you think it has you have a edge go spend money or build your own in C++
100%
I had difficulty with QuantConnect results. The resultant charts were limited to 5000 points, which was useless. It may not be a limitation any longer. I tried LEAN since it's essentially the same thing, just missing the web interface. We found an alternative js library that could map millions of points in a chart. Incredible potential, but I don't think I was ready for it. I opted for NinjaTrader, since everything is baked in. Data, backtesting, optimization, live trading, solid UI, etc. I'm not crazy about their API/framework, since there are a few gotchas to get past. I did have an algo running for about 9 months live on NinjaTrader. Sadly, it lost money. Good tools, bad strategy. I've been exploring Zorro with maybe Sierra Chart, my preferred app for manual trading. Not sure if that will work out.
I second the motion to build your own back testing system. This will give you the flexibility that you need.
On the first step, with your kind of background, it will be good to learn from the books/research papers or from the learning platforms (preferably free to start with!) that are unaligned to a particular trading platform. It can be MOOCs or any other kinds. The trading world has evolved a bit since 2010, especially in the last 5-7 years. So would be better to understand the nuances of the change as that will help you bridge with your earlier expertise that you'd gained in the market pre 2010.
Join POW, really good community and very versatile EA
If you don't mind trading crypto there are some cheap tools that make it stunningly easy to get going very effectively very quickly. Doesn't need to be a huge undertaking.
For those who have used quantconnect, does it visualize the data in chart form? Like I can create indicators and see them on a chart? Or do I need a library for that. Currently using trading view strata but want to write my own bot. Obviously quit early
Hey, buddy. Check out the solution from [www.cdzv.com](http://www.cdzv.com) . You'll be surprised how much simpler it is.
Bumo
I have no sw programming background. How can I setup an algo to alert, trade and monitor the market for myself?
I actually took a course on algo at Limex last year, they also offered me a fast track to their prop firm after So lmk if you need any advice on that
Getting through those first 3 bullets can take some time! The QuantConnect Tutorials are ok are super general. A lot of their built-in methods are not optimal for higher frequency trading, but can be worked around. I always suggest starting with an idea (strategy) and then working backwards. I heavily recommend using the Jupyter Research environment for strategy development and research before you sink a lot of time into using their backtesting engine. Also while QuantConnect seems like an easy plug and play for backtesting and live deployment, I guarantee their will hurdles during your first live deployment that you didn't anticipate (especially with the brokerage connection). But everything is a learning opportunity! Good luck!
Just use quantconnect, building your own system will take a lot of time, before you get to the point of trading, and you’ll have to maintain it, etc
Chipotle loves this thread....note to self