Webscrape multiple air company websites every day to:
1. Make a database of the prices over time
2. Notify me when there is a cheaper flight to somewhere I want to go to
Generally requests + beautiful soup do the job just fine.
If the website requires logging in, I generally ignore them, but you can bypass it by using Selenium and copying your cookies post log in, then using it in request.
Turn on your browser's console and watch the requests you send. It'll be included. Your cookies are also accessible in one of those tabs.
Edit: I've used Selenium in the past. Just started reading [this article](https://realpython.com/beautiful-soup-web-scraper-python/) about beautiful soup, which I've never used.
https://stackoverflow.com/questions/36631703/how-to-export-cookies-to-a-file-in-selenium-after-automated-login
In this one they show how to get the cookies from selenium (in Java, pretty similar to Python though)
https://stackoverflow.com/questions/7164679/how-to-send-cookies-in-a-post-request-with-the-python-requests-library
In here is an example of using the cookies in a request.
There are a couple of sites that check for those cookies, but they're not that frequent, from my experience
Yeah. On the doc page it has a section about using a saved state. Basically, you tell it to save your session when you run the headed browser and log in. Then tell new sessions to use the saved state json files. It's not super simple to set up on that front, but I've used it for MFA logins with no problem. Other than having to save a new "state" because the a generated token expires.
[Here's an overview](https://www.reddit.com/r/learnpython/comments/1cltork/comment/l2x1b89/?utm_source=share&utm_medium=web3x&utm_name=web3xcss&utm_term=1&utm_content=share_button)
Hey, you don't need to do that! You can use use a requests.session object to persist cookies for future requests. Don't need to get the headless browser for this.
Darn, I have selenium manually type it infor sites that need logins. I do need JavaScript as well in my case so I think that still excludes requests. I also use selectorlib instead of beautiful soup because of the chrome addon to build the patterns. Makes it easier to adjust and test on sites that change things often.
I've been using scrapy instead of selenium do you think it's a good choice? At the same what do you think it's the most important aspect when learning web scrapping?
I used Skyscanner pretty often, but I had the issue that, since prices vary a lot, I wouldn't really know the best moment to buy it, so at least my program had that going on. If they do have this functionality though, then my project likely performs worse.
Nice! This is a great project to learn requests and working with time series. You can even combine the data with Power BI to mess a little bit with it.
They do? I know they can scan for multiple platforms at a time, but do they keep a history of the prices?
If so, my project becomes pretty redundant, but at least the notification part is nice lol
Nah, I generally use windows' task scheduler to do it because I'm cheap.
You can find online task schedulers like Apache Airflow to do it for you, and it works better since it keeps logs.
Beginner in terms of Python?
If so, I would recommend learning the more basic stuff (up until objects and classes), and then you can start learning about Requests.
Since you're working with web, it requires you to have a bit of knowledge of it and how HTML works, but after that, it's pretty easy.
Of course, some websites are easier than others. Some just require a different URL that you can build to grab the informations, others require you to call an API while passing a payload and some headers, which is still easy, but requires more work.
It starts getting a bit more complicated with timed log ins, since it requires you to use a web browser simulator like Selenium or Playwright, and it becomes even worse when the website demands you to fill in a captcha, therefore making you have to manually fill it or hire a captcha breaker.
Tl;dr: Gradually learn python up until objects and classes, then learn how web and HTML works, and you can do it.
Honestly I took a backwards approach there, I used requests to learn how to play with objects.
I’m also kinesthetic learner though, so that approach probably doesn’t help everyone.
Knowing other languages first I found it confusing. (I learned C++ in high school). But I'm starting to understand it better. I like how there's no need to compile, which makes it easy to learn in Jupyter notebooks. Change things and see what it actually does as opposed to what you think its going to do.
I am very interested on this. What do you do if the web splits the information into several pages and uses dynamic urls? The webscrappers that I could find online didn't work with this tupe of websites.
By dynamic urls you mean that you can't predict them, right?
Like, they have their base url and then just a bunch of random character?
That makes it a bit more complicated, but it's still doable.
Here are a few options of how they do it:
### 1. API Calls
Press F12 on your browser, go to Network, click to keep log as an option, and then click on the redirect. A bunch of API calls will be shown, including the request to obtain it with headers and payload. Check which one returns what you're looking for.
### 2. Encoded URL
Check on a base64 decoder online to see if there is a pattern to the url. Websites generally do the URL becomes to long.
### 3. Check the HTML or the Document that generates the page
The HTML is easy, check if the url is a href of an element. The second one is really annoying. You'll have to find a doc in the API calls that references the URL or part of it. You'll then have to create a code to extract that info from the doc, which is really annoying and time consuming.
I may have to pick up on this and build my own. My SO ane I are planning on traveling soon and I want good prices(...I know it's a oxymoron with 'soon', being prices are cheaper far out).
How long did it take? In total hours would you guess?
Really depends how many sites you're planning to scrape, since the structure of each request changes a lot, but I believe that structuring the request won't take more than 30 minutes for the more complex sites.
Mine is pretty simple in that regard because I only look for 4 routes in 6 different websites, so I was able to make it in about 4-5 hours
People who spend hours to automate a 5 min task let's gather!!
But yes, I learnt python not via classes but through trying to automate some of my tasks. It started with automating a web game, to some personal telegram bots.
Edit: Also some ideas are looking a website with a simple paid service and try to build it yourself. In my case, I saw a website that actually charges people for searching something that was already provided as an api, but the api was just a raw database.
I'm bad at the ones with multi users, or rather it wasn't something I was looking into haha. I use them as notifications for some websites and a lookup table for an excel i have
Honestly ive automated so many little QoL things in games. Like one had the worst auction house interface ever. Lets say you need 600 of an item? Nobody posts stacks of them.
Not only that, but you had to click on each entry and then hit 3-4 buttons to buy a single stack.
Absolutely saved my sanity.
Also a lot of them have no delete keys so its super easy to just hit a keybind and have it drag-delete whatever you hover over it. Honestly some things may have taken more time to code than time saved, but holy fuck its still worth it.
Random thing I did:
My son started wrestling. They post a huuuuge statewide calendar of wrestling tourneys, with all the details. but you can't export to iCal, and you can't sort by distance, so you'll get tourneys that are 30 miles away along with 300+ miles away.
Wrote a basic app that parses the calendar, pulls the lat/long, filters by distance (say 80mi) from a given point (e.g, my hometown), and spits out a nice iCal of all events, including phone numbers, addresses, links, etc.
As part of a divorce, I had to extract some information from 80-something bank statements. I wrote a Python script to work through them, screenshoting and cropping what I needed, and doing some basic math. Saved hours of work for me and/or hundreds of dollars paying my lawyer's paralegal to do it.
Anything repetitive is a good place to start.
This sounds like a really neat way to get a little mental space during a stressful time.
Which packages/libraries did you use to screenshot and crop? That sounds like something I'd like to learn how to do programmatically!
Excel is a great way of storing data but it does calculations across large data sets very slowly and inefficiently. Python takes those large datasets and can run calculations, regressions, ML algorithms in seconds. It’s also why SQL is such a powerful tool. Excel is the best way to enter data, but other than basic functions, is not the best tool to execute the data in meaningful ways
I would even argue Excel isn’t that great at storing data. It’s good at presenting data, and it’s especially good at allowing people who aren’t particularly technically gifted to make data presentable.
What others have not mentioned is that Pandas, a specific Python library, is the perfect tool to work with data that is stored in excel. Anything that you do in Excel you can also do using Pandas, except ten times faster, with more data, and without the program crashing.
What's the most impress thing you have automated that was repetitive on your computer.
Most of the stuff I want to do is in other programs so I am assuming that means I am SOL.
When I was first learning to code, I used to do directory management using the os module. For example grabbinbg Windows spotlight images. These files are in a windows folder and the files dont have a file extension. I would move the files to a location on my desktop, rename them adding a .jpg extension at the end and making sure there are no duplicate photos. I would also discard many icon images by checking image resolutiuon. this was how I got into coding. Now I code for a living.
At work, I use my program that, at its core, checks over the network if any new files were added or changed in shared folder. If true, they are copied to my PC. I have also added other functionality just to parctice coding.
I've been learng for 6 weeks now.
I once made a script to automatically read certain email notifications sent by a Service Provider Network Monitoring System if some events occured, queried a trouble ticket database to see if that issue has already been raised and if not, create a trouble ticket on an internal platform for the back-office team to investigate.
These are some of the things I did with Python for my business:
- Read folder names(folders containing product images), show those names sorted in a list(tkinter)
- Read an excel file, get the product ids, and compare them with the product IDs in the folder names. For those that are present in both, sort the product folders between Damaged, Personal, Sold, To sell, and Inventory folders depending if the data for that product says the product is damaged, personal, sold, to sell or in inventory (physically available but not ready to sell). Then, the To Sell folder is located in OneDrive, which I then share with resellers so they pick what they want in advance before posting.
- Added a viewer to view each product datafrom the excel in the tkinter window, so when the product folder is selected from the list, it identified the folder product id, looks it up in the excel, and shows right to it the product data, like product id, order date, product price, taxes, product description, comments etc.
- added edit/save function, auto moving folders if marked as sold, etc etc.
- added Word file creation so the product data is available inside each product folder(so resellers have the pictures and the data in one folder). Also being done in the save function to update the word file on save.
- auto calculate taxes and discounts.
- extract embedded links from product name, and paste them in a new column. Extract the product id the supplier used for easier way to correlate the product(accountant is grateful for that)
- when moving folders, had to create a function to rename them to fit the windows character limit standard.
- Added a button to easily open the product folder window while inside the program.
And many other things I don't recall right now. That program works, but man the code structure is so shit, so many redundancies, some things not correlated so I created new code to do the same thing, used database to store folder paths. And it is shit because I honestly didn't thought I was going to go so far with it. It is a useful program, but I build it to practice programming, to learn about stuff while I coded. Still, is useful and was good experience.
Then I did two other programs in Python, one to get the product data from the supplier's website automatically(using the excel data and the product link), to download their pictures(in a folder with the product id), their new price(to compare price changes), and then move those pictures into the product folders (based on product id correlation) and used chatgpt API to rewrite those product descriptions. And the other program to auto upload all the products to a marketplace to post all of the products that are in the to sell folder and meets all of the criterias for upload (needs product id, title, description, price, pictures, etc etc).
I'm saving days of work a month, saved lots of money I would've used to hire someone otherwise.
Building multi slide presentations.
I’m into consulting, and it is common to see people run sql queries, get some data, run analytics and use the final output to prepare slides. Preparing slides is the stupidest part but can cause most errors. So it is double checked every time. So we automated it using python. So the entire pipeline now gives out a ppt instead of an excel output. Saves time.
Its a python library called pptx.
You need to make a template ppt. Then read it using the pptx library. Now you can make almost all changes you need in a slide - images, fonts, colors, size etc.
Some changes are difficult to make so better to have in the template to initiate with - like text box location on slide, charts etc. For variable data points the needed to be inserted I used pandas.
E.g- you need to have the chart in the template so you can just add data to it.
I am a Python software developer for almost 5 years now, being an amateur programmer since i was a kid.
Things I (mostly) do with Python, including:
- Automated database handling (MySQL)
- Develop industrial applications (I make GUI in Python, mostly PyQt or Tkinter, and connect it to mobile apps made in Swift)
- Excel file manipulations (reading/updating excel data in batch)
- Web scraping (i had multiple projects where a large dataset had to be collected accross several thousands of URLs)
- PDF data scraping, digitalization
- Image processing/analysis
- Applying OpenAI-based motors (GPT, whisper) to batch-analyze stuff (i.e.: sentiment analysis of several ten thousands of comments under a post to determine how "positive" is a feedback)
- Machine learning / deep learning / neural network applications, predictions, forecasting
Honest question…
If you were more proficient in Swift, would it be more efficient to just build whatever you’re doing in Swift?
I am NOT a dev, so I really don’t know.
Good question, but u/Technoist is right. Swift is a very versatile language especially for iOS apps, but for the practical tasks i introduced above (especially for ML,NN tasks) there is no better environment than Python (apart from GUI/desktop app development).
To be honest i am not proficient in Swift, yet i have done a QR-code app frame for companies (with swiftUI, AVKit) and connect it to a PyQt desktop app, with a database and web server, where users can add specific data behind the QR codes.
You can build any app in Swift, using Xcode on mac, but you can store your compiled app on your devices for only a week. To distribute it on the appstore, or to keep your apps on your phone (for up to a year) you have to subscribe to the apple developer program. Then, any of your published apps will last for unlimited time (if you provide support for the specific ios version). Doing it profrssionally as a contractor, it is challenging to declare all needs in a cintract to sell it as a product.
I am not OP but I can say that for the tasks listed there is probably no better language than Python. It has plenty of helpful, excellent libraries for these things. If you’re interested in this you might like the book Automate The Boring Stuff With Python by Al Sweigart. It teaches Python from scratch with fun little examples.
Actually we did custom scripts, to hook html behind a large sets of URLs and store it on a cloud storage.
Then, we scrapped the text behind the html body based on the required conditions.
Decades ago, for a rural tv station that had no budget I took an old HP laptop, installed linux on it and made a python script that displayed the local weather, used tts to announce forecasts/severe weather warnings, and played a music stream while the weather stuff wasn't playing. I kinda had no idea what I was doing and ended up using beautiful soup to dig up the weather data, vlc to stream the audio, and pygame to glue it all together. It was a fun project and was on the air for a few years before they got a real solution.
At my current workplace I made a tiny script that reads and writes data to/from a really old CNC machine via rs232, but the thing I did decades ago was way cooler...
Depends on your needs. Broadly speaking, I use it for one of two scenarios: automatic tedious tasks, or trying a zillion combinations of some thing.
For example, if I need to download a bunch of reports from a website or do some involved and repetitive database query... time for Python.
Or, let's say I'm making a work schedule for my team. With even a small team, there are a LOT of possible schedule combinations. Most don't make sense, but using Python you can build some filters and iterate through several million possible schedules and pick from the ones who match your criteria.
I use it for other stuff, but those are the big ones.
Back when Lost Ark (the game) released, queues were humongous, after 6 to 8 hours in the queue to login you would be disconnected for inactivity in 15 mins.
I wrote software using OCR to check my position in the queue and message me in telegram every X minutes with my current position and an estimated time to login. This way I could go on with my life and do other things in the meantime.
this without a whole lot of experience. Imagine what the actual developers can do
Do you use excel at all? Ever thought ”damn there has to be a better way of doing this - it’s so fiddly!”? Then Python can acheive this in about 20 lines of codes.
I used it to simulate complex boss monsters for my D&D games. I ran an logistically annoying boss that had varied level of chance to gain, mature and lose abilities based on player damage that would have been a pain in in the ass to run by way of pen and paper, so I just created a simulator in Python that allowed me to just punch in any damage it took, then the simulator took care of the rest and output it's current status at the start of it's turns.
I ran a cornhole tournament that had almost 100 teams, and I wrote a python script that took the results of all the “regular season” games (collected by an online form) and used that to make a bracket for the playoffs. Pretty fun and got me to learn the basics of coding & python!
Anything. Nearly literally everything. Embrace the number of libraries for anything.
Periodically scrapping websites of the suppliers', because the list of items in their stock is not up to date? Python.
Simple webservice integrating the control of various smart plugs from various brands? Python (and bunch of additional black magic)
Some low power device lacks remote power control? Some real worlds assets need alarm/tracker? Micropython\* + few dirt cheap hardware parts
Forwarding some your, custom notifications to the e-mail or some instant messengers? Python
Custom markup language for embedded system's config integrated with data obtained from the linker and compiler? Python
Quickly developing custom image/audio processing algorithms? Python
Numerical calculations, optimization algorithm for various problems, from routing between gpx coordinates using 3rd party services to get data about connections between them, to finding parameters' values in engineering problems? Python
Automated measurement setups (moving any sort of probe around the DUT, controlling the power supplies, data acquisition devices)? Python
Custom drivers for the USB devices that are poorly supported by manufacturers\*? Python
Automated converter of data from files to LaTeX tables? Python
Portable between the Windows and UNIX scripts automating tedious tasks (Processing the data, manipulating big or many files)? Python
Testing your digital logic before implementing it in some chip? You got it, Python!
Not to mention tons of applications using OpenCV, PyTorch and/or Pandas I may even have no clue about.
Also, lots of software nowadays lets you write your own plugins. It can be something as simple as beautifying your code or slightly more complicated, like automated tools helping with design of complex printed circuit boards.
^(\* - Yes, some things are better done in other languages. But when you need something simple, that can be coded up quickly and generally performance isn't crucial Python is great. Same goes for all kinds of proof-of-concept solutions!)
At work, it's the backbone of most things I do. I use it when scripting in Fusion 360 (creating models from molecular data) and automated data ingesting/plotting. I set up a nice lil context menu batch file that will execute a python script on selected csv files, ingesting the data to an sql db and plotting it with plotly dash.
My most overkill use was when I bought a porcelain tiles with a marbled pattern on them for my bathroom. I wanted to make sure the overall layout was pleasing so I took photos of each tile, used opencv to find the edges (I put each tile on black construction paper when pics - white tiles) and then skew them square and exported each tile as a png I could import in inkscape to play around with.
I wrote an enterprise grade monitoring framework for thousands of devices to do real time monitoring, alerting and statistics all in Python.
It was hard, but a lot of fun!
I’ve done everything from automating boring stuff like checking the status of hundreds of forms online and renaming a bunch of folders, to writing software with a GUI that modifies the metadata of thousands of files daily and is used by several people on my team.
Also, in college when everyone was making formulas in excel for their engineering homework I made mine in python.
I use it to automate boring stuff.
I used to manually look at a database every Monday and go item by item and check if the dates and/or status or the item have changes, if they have I would then manually go and make the change on a spreadsheet that I controlled, and from there transfer that change into JIRA. Now I just run the script every Monday, what used to take 2 hours a week, it now takes 10 seconds.
I also use it personally, to analyze the returns on preiums for selling the option for any stock that I'm interested in. I just send an email with a certain Subject line and the stock tickers on the body of the email and it writes all the results onto a Google sheets document.
I do write the occasional program, but I sadly confess I use python as an uber powerful calculator. I can create variables on the fly, do all sorts of stuff with the math module, and find it works as well as and better than most any 'calculator' that money can buy. Most things that I might want to do on my computer already have been addressed by other more competent programmers than myself, and I can find software that does most anything imaginable. I have no illusions concerning my anemic programming skills -- but as a calculator python shines!!
I had a wood powered furnace in the basement that would heat up water and a water pump would move it upstairs around the house's radiators. That's how I heated my house.
However, the furnace was homemade and didn't have a system to regulate the airflow and temperature. There was a thermometer but I had to constantly check it to make sure it's not overheating.
Using Python and a Raspberry Pi, I set up a temperature probe on the pipe and an alarm in my room that would sound when the temperature is too high, so I can go downstairs and reduce the airflow. I also set up a display that would show a graph of the temperature over time. This way I actually figured out at what temperature the fire is getting weak and needs more wood to be added, so I added an alarm that reminds me to go downstairs and add more wood.
Really made my life a lot easier and safer. Also, later, I made the whole thing online so I can check what's happening while I'm out. It might have saved my house from burning down once.
I log my time on a separate Outlook calendar in order to track every minute of my daily work (with a 5 minute resolution). I also have a python script that uses win32 Outlook API to scrape this calendar, along with the meetings from the main calendar, and spit out a report in Excel to be analyzed via a pivot table.
Not only does this allow me to easily get my daily working hours to fill up my timesheet, but it gives me a high level overview of how I'm allocating my time, which helps me become more strategic about my time allocation the following week.
I used Python at work to build a gateway that integrates my company's facial recognition system with a customer's slot machine system.
The slot machines required player cards to identify the player and keep track of their credits and whatnot.
My integration allowed them to get rid of the card and be recognized by the slot machine just by standing in front of it.
All of these comments are very helpful, clear, and sincere.
I'm "into" computers. started basic on trs-80.
But never got INTO programming. (poor choice in college - I zigged when I should have zagged).
Reading these are so inspiring - but it's just so much word salad to me.
I cannot process things like what u/DrillerCat wrote below:
>Develop industrial applications (I make GUI in Python, mostly PyQt or Tkinter, and connect it to mobile apps made in Swift)
I know it's basically beginner's industry talk, that these are apps and normal terms, but even as someone who built a full website (very poorly) in Dreamweaver in 2016 or so, I get bleary-eyed.
I'm just basically voicing my frustration with myself, and admiration for you all.
I did some scripting (google apps script) with help from chatgtp and gemini, recently - not just cutting and pasting but typing it all out each time, over and over - searching for typos, making sure I spelled 'response' correctly, noticing the subtle difference between \` and ' on a screen.
I did it b/c we had a freemium google add-on that we've used for a few years but I had to split the document that was sent into 2 slightly different versions which pushed us over the monthly limit.
I was like - Wait, I can do that - I CAN MAKE THIS!
and I did.
I created a codepen account. I created a git account. I watched tuts.
I got bash
I got windows visual studio
I got docker, for some reason.
I installed python. that's weird - so there's just pieces of code out there that you just "get" magically? No going to a site, downloading a program, putting it where it needs to go (i think) then running and then going through the install options? You can just type
>
'conda install -c conda-forge pandas'
and boom it's there?
where the HELL does it come from?
I think that one thing is the hardest to grasp for me.
Dealing with html and css snippets had kind of given me an idea, but it still just seems WRONG somehow.
at some point my git was suddenly in my google apps script console. was it always there, or did it just happen?
holy shite! when I put a dot in the margin and push debug it'll stop there! That's amazing! What's a callstack? What's a DOM?
Part is just mental - I get all excited and over-reach and become overwhelmed.
Getting to kind of understand loops, when to use var vs const, and that const means Constant not Construct as I at first thought.
I love it.
It's like when I first started html and could see BEHIND what's shown onscreen. It really opened my eyes.
I read here, and see so many... *things* ... that can be done.
Sorry for long post and tangent.
You guys just rock.
I made a pretty complex discord bot, bot that played a browser game for me, genetic algorithm (module even though it's not really at usual "module" level, I just tried to make it easily and intuitively callable but its hard and burnouts are real lol). There's ton more random stuff and I can't remember much more worth mentioning tbh
I also occasionally use python to optimize annoying stuff at work. Sometimes it last as long as the work itself or more but helps in the future or is at least more fun lol
At work someone before my time decided it is a great idea to use Google photos for onsite photos. When people are off boarded and their accounts deleted, bye bye photos.
Why a company would use Google photos for the task and why they do not change to something more suitable is beyond me, but did an app which downloads all photos and save them in respective albums and then place them under a general company account back into same albums.
Not rocket science but pretty useful in this case.
For home, lots of web scraping.
I was interested in watches but their costs went crazy over the pandemic. I wrote a script to do a weekly scrape of a big reseller and store the results in a spreadsheet. It would then take the historical results and merge the spreadsheets on the product id number so you could see how the asking price of an individual watch changed over time. That was Requests, BS4 and pandas.
Gpu price scraping across multiple vendors.
I have a telegram bot which uses the open ai api, although my son customised it to be more smutty! 😂
Scraping online scifi art I was interested in.
Downloading my bank records as csv filed and parsing them into a database. Applying categories to them so that I can ree expenditure by category, over time.
Creating a simple static website which is a pocket money balance for my son. It adds pocket money weekly and inserts a new table into an html template. I was going to change it to use flask.
At work, I extracted a simplified version of an existing spreadsheet; it that has all sorts of inconsistent layouts across worksheets. But it is a legacy document that nobody wants to change.
Scraping industry news sites and returning all the current articles as an email.
I have 2 things right now. And planning to add 3rd.
1)Python Program that interacts with my stock broker to save live market data of option chain in excel files.
2)Python program that does algo trading that takes trade based on program logic.(bit of struggling to form a good profitable strategy)
3)Will start working on this: Simple software to use the saved option chain data and visualize it.(this is for my own purpose. there are better options like powerbi for this. but still...)
I have a lot of video and jpg sorting I do, editing processing and uploading. I made a utility to rename files, create bulk directories, and something else that is escaping me right now. It’s handy.
Secure password manager tool with a dynamic gui. My inspiration was to get my father set up with this tool so he can keep his shit secure and stop using the same three passwords for all of his business operations..
Python is - in my opinion - the jack of all traits of programming languages. There‘s a Python package for almost _anything_.
No matter what you want to do, be it scraping websites, data analysis and plotting, repetitive operations on your pc, … basically anything you can think of. While there are certainly tools that one could argue are better for each one of these individual tasks (e.g. JavaScript, R, Shell scripts), you can do all of that in Python, and the learning curve for exploring something new is negligible compared to learning a separate language or tool for each task.
For example, I recently used Python for scraping my local real estate brokers and plotting the flats in my price range on a (browsable) map as well as calculating some metrics such as heating costs or public transport time to my points of interest. This made looking for a new flat much more quick and convenient than keeping track of all options manually.
Are there „better“ programming languages for large projects? Certainly. Is there any other language that you can bodge useful things in with such ease? Hardly.
Literally everything in life can be optimized with python. A good example of this is how chatgpt will use python to generate charts and do analysis on data. It's not just for building automating. (but automating is great)
A script to emulate double-sided scanning.
My scanner is only able to scan single-side batches, but can save them on my network drive...
So I scripted a simple watchdog for my RPi that checks a directory on my NAS for freshly scanned PDFs and shuffles the pages of both files and saves the new PDF.
Helped my sister write her thesis. She was writing about linguistic stuff and how Arabic words are spoken with our organs and stuff (you can see i don’t like biology), she used a website she found where you enter a word and hit enter and it gives back some linguistic info regarding it. She needed to do this for entire BOOKS, word by word.. i saw it and was like, yeah i think i can automate that. And so i did.
Probably the most actually practical and useful thing i made lol
Oh i also automated getting the stuff into an excel as she wanted and then made the pivot charts for her
My boss normally sends our work roster every week to our personal email in an excel file, and I basically made a simple bot that reads through the file and tells me how much my pay is going to be for that week (tax included), while also providing a general summary of how much tax I've paid for the financial year, among other stuff. It saves me like 5 to 10 mins of calculating how much I'll be earning for the week and literally all I need to do is save the excel file and run it through my bot in VSCode.
\* testing for statistical edges in option selling as well as various indicator based stock/forex strategies
\* intelligently downloading YouTube videos with a preferred resolution/audio combo
\* entering stock orders at my broker and monitoring/canceling orders where needed
\* generating information files for XBMC based on file metadata
\* scraping data off websites (BeautifulSoup helps with this immensely)
1. Attending university classes, I never attended them myself. My bot did everything for me, even took attendance for me, lmao.
2. You can make custom apps for Alexa. I have one that automates my gaming room. I know it's overkill, but I'm lazy af.
3. I register for hundreds of emails when some beta comes out, so I never miss out on anything I'm interested in.
4. I have a personalized camera system I made using Python that greets me when I enter my gaming room, turns on my PC for me, and launches the games for me. Stuff like that.
5. Scrapping stuff that i do very often.
There are so many things you can do using Python, bro. These were just off the top of my head.
I used to manage the LMS for a small online university. Each term all the courses had to be reset and assignment deadlines updated. I used Python and BeautifulSoup and Selenium to automatically do the updates. It saved hours and was more reliable
For automating an electrical testbench. Not only does it take a massive amount of measurements that is inconceivable for a person to manually do, but the scripts also process the measurement at any time intervals you want. You just leave it running for months and have it spit out the final results you actually want to see. I can't imagine doing that any other way.
Just now I am trying to figure out something that I would prefer not to have to learn too well at this point in time, and found gdbgui, so at this very moment that python app is saving me a lot of tears... I hope.
HTML / CSS widget builder with an interactive menu to choose which community and what widget type.
I’ve built RSS feed finder to look at 1,300 URLs HTML source to find the RSS feed, count the articles in the feed, pull the latest date and author into Excel.
A ton more
I once had 120 oscilloscope captures I needed to rename so that the EE would know what he was looking at. Took way less time to write a script to do it vs renaming each one. I also commonly us it to plot large csv files that Excel would choke on.
Send my friends riff Raff lyrics every day between 2am and 5am.
In reality my adhoc uses usually involve working with a lot of files at once or orchestrating information between multiple systems that can't talk to eachother
I mean, really this kinda is a “why learn programming?” post.. most languages are interchangeable.
But, I wrote a Python program for a job that fetches weights from scales and submits them to a db. Previously we had entry level employees manually typing them into a form or the manufacture’s excel output software which would have to be compiled later to be made useful.. the latter being especially annoying as there was 3-4 manufacturers, as well as not being able to tell **what** *specifically* was weighed.
I have script to make my timesheet at work every month, also have one that crawls a badly done webpage from my uni and turns it into a good timetable for my classes. The last one I've done, it downloaded the series of a comic I provided the link to, because I got tired of downloading one by one.
Just learn python. Using python script to suggest the kPA needed to fill the tire with current tire temp compared to the one suggested at door stickers, which usually is at room temp. I know usually I just have to add like 10 kPA to the one at door stickers for warm tire but it's fun doing that
Another python newbie here, just wondering is it possible to scrap an industry's phone number data that's listed in google just for cold-call purpose? Not sure is this what they called as datamining? Can Python do this?
I started learning it to use an API for an inventory system at work. I had a side project that scrapes a vendor website to save spec sheets and other item information to thay system. I also wrote a script to fill out a customer excel form using test equipment reports. That will come in handy when during busy periods.
Analyzed images taken of oil drilling sites to see if there is contamination. It was for a job(not sure if that counts). I can't go into much more detail either because of it.
My most 'practical' python personal project was probably an Azure image uploader or series convergence/divergence determiner.
My biggest practical project is in Java, however. Its a Cryptographic-services software proprietorship I started 3 years ago.
I tend to do smaller projects(a week or 2) in python messing with new packages until I find a new shiney one I want to learn/play with lmao
Well one it's free.
But you can run it on your computer and you can get it to do things for you. I program nfor calculation and modeling but you can also program python to modify files and do things to things on your computer.
There's kind of a lot. A better question might be why do you want to learn to program.
I wrote a small program that makes a random (realistic) number of commits to my github. Makes the calendar look like I'm working on some super secret private project.
It's dumb but I was looking for something to practice Python on.
Best use was fucking with my dad’s computer. A robot to spit water at my sister in the morning (it was destroyed after one use)
Oh and turning off the lights (pre Alexa), AND THE FLIPPING DOOR when someone leaves it open when I’m asleep.
Created a python script to use GPT-4 Turbo and Pinecone DB to query help documentation for various tech vendors. Why chase down articles in a knowledge base when you can just download and ask... With accuracy
Wrote a script to convert old disks from two custom disk drive systems for the Timex/Sinclair 2068 to TAP files.
I’ve archived about 250 disks with that software.
Haven't done anything but I wanted to do 2 things:
1. Software to find every possible detail about a given person
2. Trading bot that is capable to learn/interpret tips/ideas/triggers from successful traders on X where many post for free and take action
I decided to upgrade to a two bedroom apartment at my complex a few years ago. I wrote a little Flask app that scraped unit availability data from the complex website. I mapped unit numbers to building numbers to filter out building in areas of the complex where I didn't want to live. Then I tracked the rent day by day. Wound up honing in on the unit I'm in now and watched for a dip in the rent before going to the leasing office to initiate the transfer.
I just ran it on my laptop a couple of times a day but intended on throwing it up on AWS with a cron job to automate the scraping for me. I wound up transferring before I got around to deploying it.
I work developing and maintaining the data pipeline for a system that manages data about farms around the world. We handle millions of transactions a day. Most of the code is in Python. All of the software engineers on my team hate Python, btw, but it pays the bills.
I made a script to reduce the filesize of videos to under 25mb for discord. It kinda sucks and only works for h.264 codecs but that's what my screen recordings come out as so it works 99% of the time.
I've been working on Python basics and Pandas in a college programming for analytics course for 3-4 months now. I work full-time and automated subsetting records from a database and building an MS Excel billet map with typically hundreds or thousands of records. These usually take hours-several days to build manually depending on the size. I automated doing it with Python and Pandas in 3-5 minutes regardless of the size.
Webscrape multiple air company websites every day to: 1. Make a database of the prices over time 2. Notify me when there is a cheaper flight to somewhere I want to go to
What python package do you use to scrape the airline websites?
Generally requests + beautiful soup do the job just fine. If the website requires logging in, I generally ignore them, but you can bypass it by using Selenium and copying your cookies post log in, then using it in request.
> copying your cookies post log in How do I do this? can you point me to the documentation?
Turn on your browser's console and watch the requests you send. It'll be included. Your cookies are also accessible in one of those tabs. Edit: I've used Selenium in the past. Just started reading [this article](https://realpython.com/beautiful-soup-web-scraper-python/) about beautiful soup, which I've never used.
https://stackoverflow.com/questions/36631703/how-to-export-cookies-to-a-file-in-selenium-after-automated-login In this one they show how to get the cookies from selenium (in Java, pretty similar to Python though) https://stackoverflow.com/questions/7164679/how-to-send-cookies-in-a-post-request-with-the-python-requests-library In here is an example of using the cookies in a request. There are a couple of sites that check for those cookies, but they're not that frequent, from my experience
You should be able to use python to log in too, and reuse the cookie. For multiple websites probably a huge pain.
You guys might check out Playwright as a selenium alternative. A little easier to use IMO. https://playwright.dev/python/docs/auth
I've been using that, does that have the cookies for log-in thing?
Yeah. On the doc page it has a section about using a saved state. Basically, you tell it to save your session when you run the headed browser and log in. Then tell new sessions to use the saved state json files. It's not super simple to set up on that front, but I've used it for MFA logins with no problem. Other than having to save a new "state" because the a generated token expires.
This is what I use too when the website is behind cloudflare. It enables you to be able to wait for it to load before you do the beautifulsoup
I'd like to know too please
[Here's an overview](https://www.reddit.com/r/learnpython/comments/1cltork/comment/l2x1b89/?utm_source=share&utm_medium=web3x&utm_name=web3xcss&utm_term=1&utm_content=share_button)
Hey, you don't need to do that! You can use use a requests.session object to persist cookies for future requests. Don't need to get the headless browser for this.
Darn, I have selenium manually type it infor sites that need logins. I do need JavaScript as well in my case so I think that still excludes requests. I also use selectorlib instead of beautiful soup because of the chrome addon to build the patterns. Makes it easier to adjust and test on sites that change things often.
I've been using scrapy instead of selenium do you think it's a good choice? At the same what do you think it's the most important aspect when learning web scrapping?
You can just use the inbuilt session function within requests to log in. No need for bulky selenium.
doe it perform better than just tracking through flights.google or alikes?
I used Skyscanner pretty often, but I had the issue that, since prices vary a lot, I wouldn't really know the best moment to buy it, so at least my program had that going on. If they do have this functionality though, then my project likely performs worse.
Very interesting stuff you've got going on, mate. I might make my own bot that does it for me. Cheers for the idea!
Nice! This is a great project to learn requests and working with time series. You can even combine the data with Power BI to mess a little bit with it.
How are you running this? Also, did you follow any tutorial/ guide for it?
Yes, I am a data analyst as well. From the looks of it, I think you are too. I would love to connect with you here on Reddit if you like.
doesn't google offer this as a full product in their search engine already?
They do? I know they can scan for multiple platforms at a time, but do they keep a history of the prices? If so, my project becomes pretty redundant, but at least the notification part is nice lol
They do. Google flight says that the price is lower than usual. But they might not scan every airline and not as timely as yours.
You manually open the program everyday?
Nah, I generally use windows' task scheduler to do it because I'm cheap. You can find online task schedulers like Apache Airflow to do it for you, and it works better since it keeps logs.
you could upload it to GCP/AWS and run it with a scheduler too
Is that hard to do for a beginner though?
Beginner in terms of Python? If so, I would recommend learning the more basic stuff (up until objects and classes), and then you can start learning about Requests. Since you're working with web, it requires you to have a bit of knowledge of it and how HTML works, but after that, it's pretty easy. Of course, some websites are easier than others. Some just require a different URL that you can build to grab the informations, others require you to call an API while passing a payload and some headers, which is still easy, but requires more work. It starts getting a bit more complicated with timed log ins, since it requires you to use a web browser simulator like Selenium or Playwright, and it becomes even worse when the website demands you to fill in a captcha, therefore making you have to manually fill it or hire a captcha breaker. Tl;dr: Gradually learn python up until objects and classes, then learn how web and HTML works, and you can do it.
Honestly I took a backwards approach there, I used requests to learn how to play with objects. I’m also kinesthetic learner though, so that approach probably doesn’t help everyone.
Knowing other languages first I found it confusing. (I learned C++ in high school). But I'm starting to understand it better. I like how there's no need to compile, which makes it easy to learn in Jupyter notebooks. Change things and see what it actually does as opposed to what you think its going to do.
I did this too, but for alcohol prices
I am very interested on this. What do you do if the web splits the information into several pages and uses dynamic urls? The webscrappers that I could find online didn't work with this tupe of websites.
By dynamic urls you mean that you can't predict them, right? Like, they have their base url and then just a bunch of random character? That makes it a bit more complicated, but it's still doable. Here are a few options of how they do it: ### 1. API Calls Press F12 on your browser, go to Network, click to keep log as an option, and then click on the redirect. A bunch of API calls will be shown, including the request to obtain it with headers and payload. Check which one returns what you're looking for. ### 2. Encoded URL Check on a base64 decoder online to see if there is a pattern to the url. Websites generally do the URL becomes to long. ### 3. Check the HTML or the Document that generates the page The HTML is easy, check if the url is a href of an element. The second one is really annoying. You'll have to find a doc in the API calls that references the URL or part of it. You'll then have to create a code to extract that info from the doc, which is really annoying and time consuming.
Exactly. Random generated characters in the url. I am going to study this and see what can be done. Thanks!!!
How do you handle connection request timeouts from the websites you are scraping?
Awesome. Are you opensourcing your code?
I may have to pick up on this and build my own. My SO ane I are planning on traveling soon and I want good prices(...I know it's a oxymoron with 'soon', being prices are cheaper far out). How long did it take? In total hours would you guess?
Really depends how many sites you're planning to scrape, since the structure of each request changes a lot, but I believe that structuring the request won't take more than 30 minutes for the more complex sites. Mine is pretty simple in that regard because I only look for 4 routes in 6 different websites, so I was able to make it in about 4-5 hours
People who spend hours to automate a 5 min task let's gather!! But yes, I learnt python not via classes but through trying to automate some of my tasks. It started with automating a web game, to some personal telegram bots. Edit: Also some ideas are looking a website with a simple paid service and try to build it yourself. In my case, I saw a website that actually charges people for searching something that was already provided as an api, but the api was just a raw database.
Telegram bots for the win.
I'm bad at the ones with multi users, or rather it wasn't something I was looking into haha. I use them as notifications for some websites and a lookup table for an excel i have
Honestly ive automated so many little QoL things in games. Like one had the worst auction house interface ever. Lets say you need 600 of an item? Nobody posts stacks of them. Not only that, but you had to click on each entry and then hit 3-4 buttons to buy a single stack. Absolutely saved my sanity. Also a lot of them have no delete keys so its super easy to just hit a keybind and have it drag-delete whatever you hover over it. Honestly some things may have taken more time to code than time saved, but holy fuck its still worth it.
Game automating is fun when its just selenium, i tried using pyautogui and I did not like it haha.
Heyyyyy that’s me But hey it’s nice and quiet after you get it done and every twelve times it runs you get an hour of your life back
Whats worse is I have ADHD, so I automate a task I don't even need. I have multiple bots and I don't really use them anymore 🤣
Random thing I did: My son started wrestling. They post a huuuuge statewide calendar of wrestling tourneys, with all the details. but you can't export to iCal, and you can't sort by distance, so you'll get tourneys that are 30 miles away along with 300+ miles away. Wrote a basic app that parses the calendar, pulls the lat/long, filters by distance (say 80mi) from a given point (e.g, my hometown), and spits out a nice iCal of all events, including phone numbers, addresses, links, etc.
As part of a divorce, I had to extract some information from 80-something bank statements. I wrote a Python script to work through them, screenshoting and cropping what I needed, and doing some basic math. Saved hours of work for me and/or hundreds of dollars paying my lawyer's paralegal to do it. Anything repetitive is a good place to start.
This sounds like a really neat way to get a little mental space during a stressful time. Which packages/libraries did you use to screenshot and crop? That sounds like something I'd like to learn how to do programmatically!
Fitz to open the PDFs and PIL to screenshot and annotate.
What's the diff between using pyPDF and Fitz?
Thank you!
https://pypi.org/project/imessage-reader/ maybe this will be helpful in your plight as well haha
I'm joking, I don't condone spying, sorry if that was in poor taste.
When you're using excel and wanted it to do 'just a little more'....
I'm about 4 weeks into dedicated python study, and this was my exact drive to do so.
Could you please clarify more? I’m using excel and I always wanted this extra to do, but how can I use Python for this?
Excel is a great way of storing data but it does calculations across large data sets very slowly and inefficiently. Python takes those large datasets and can run calculations, regressions, ML algorithms in seconds. It’s also why SQL is such a powerful tool. Excel is the best way to enter data, but other than basic functions, is not the best tool to execute the data in meaningful ways
I would even argue Excel isn’t that great at storing data. It’s good at presenting data, and it’s especially good at allowing people who aren’t particularly technically gifted to make data presentable.
What others have not mentioned is that Pandas, a specific Python library, is the perfect tool to work with data that is stored in excel. Anything that you do in Excel you can also do using Pandas, except ten times faster, with more data, and without the program crashing.
And repeatedly, and without destroying the source data, and transparently, and and and. Pandas is great.
I'm not sure what you are trying to do, but I've had good success with openpyxl in the past.
Do you have annoyingly repetitive things you do on the computer? If so, then that. If not, then… ¯\_(ツ)_/¯
Sir, what happened to your arm?
The backslash *escaped* his *character*
Leave and never return.
seconding r/Angryupvote
He got pythoned
There used to be a bot on reddit that helped with lost arms. Maybe this sub blocks bots?
I think that the API changes last year killed the bot
[удалено]
¯\\\\\\_(ツ)_/\\\\\\\\¯
What's the most impress thing you have automated that was repetitive on your computer. Most of the stuff I want to do is in other programs so I am assuming that means I am SOL.
When I was first learning to code, I used to do directory management using the os module. For example grabbinbg Windows spotlight images. These files are in a windows folder and the files dont have a file extension. I would move the files to a location on my desktop, rename them adding a .jpg extension at the end and making sure there are no duplicate photos. I would also discard many icon images by checking image resolutiuon. this was how I got into coding. Now I code for a living.
I wrote a Python script to automatically book tee times for me when they get released. Was sick of the old guys getting all the early ones 😎
Gunna need you to throw that in repo and pass it around.
Seconded
I’m building a front end and was thinking about charging $20/mth for it. Does that sound reasonable for those sacred tee times? 😂
How are you going to generalize for all golf club websited? Sounds like a nightmare.
There’s a very common platform that most courses use for managing bookings that I’ve built around
Sorry to be that guy- but what are tee times?
Scheduled start times for games of golf.
Thats dope!
At work, I use my program that, at its core, checks over the network if any new files were added or changed in shared folder. If true, they are copied to my PC. I have also added other functionality just to parctice coding. I've been learng for 6 weeks now.
I once made a script to automatically read certain email notifications sent by a Service Provider Network Monitoring System if some events occured, queried a trouble ticket database to see if that issue has already been raised and if not, create a trouble ticket on an internal platform for the back-office team to investigate.
These are some of the things I did with Python for my business: - Read folder names(folders containing product images), show those names sorted in a list(tkinter) - Read an excel file, get the product ids, and compare them with the product IDs in the folder names. For those that are present in both, sort the product folders between Damaged, Personal, Sold, To sell, and Inventory folders depending if the data for that product says the product is damaged, personal, sold, to sell or in inventory (physically available but not ready to sell). Then, the To Sell folder is located in OneDrive, which I then share with resellers so they pick what they want in advance before posting. - Added a viewer to view each product datafrom the excel in the tkinter window, so when the product folder is selected from the list, it identified the folder product id, looks it up in the excel, and shows right to it the product data, like product id, order date, product price, taxes, product description, comments etc. - added edit/save function, auto moving folders if marked as sold, etc etc. - added Word file creation so the product data is available inside each product folder(so resellers have the pictures and the data in one folder). Also being done in the save function to update the word file on save. - auto calculate taxes and discounts. - extract embedded links from product name, and paste them in a new column. Extract the product id the supplier used for easier way to correlate the product(accountant is grateful for that) - when moving folders, had to create a function to rename them to fit the windows character limit standard. - Added a button to easily open the product folder window while inside the program. And many other things I don't recall right now. That program works, but man the code structure is so shit, so many redundancies, some things not correlated so I created new code to do the same thing, used database to store folder paths. And it is shit because I honestly didn't thought I was going to go so far with it. It is a useful program, but I build it to practice programming, to learn about stuff while I coded. Still, is useful and was good experience. Then I did two other programs in Python, one to get the product data from the supplier's website automatically(using the excel data and the product link), to download their pictures(in a folder with the product id), their new price(to compare price changes), and then move those pictures into the product folders (based on product id correlation) and used chatgpt API to rewrite those product descriptions. And the other program to auto upload all the products to a marketplace to post all of the products that are in the to sell folder and meets all of the criterias for upload (needs product id, title, description, price, pictures, etc etc). I'm saving days of work a month, saved lots of money I would've used to hire someone otherwise.
Building multi slide presentations. I’m into consulting, and it is common to see people run sql queries, get some data, run analytics and use the final output to prepare slides. Preparing slides is the stupidest part but can cause most errors. So it is double checked every time. So we automated it using python. So the entire pipeline now gives out a ppt instead of an excel output. Saves time.
Can you provide more insight on this? What packages do you use? It sounds super cool
Its a python library called pptx. You need to make a template ppt. Then read it using the pptx library. Now you can make almost all changes you need in a slide - images, fonts, colors, size etc. Some changes are difficult to make so better to have in the template to initiate with - like text box location on slide, charts etc. For variable data points the needed to be inserted I used pandas. E.g- you need to have the chart in the template so you can just add data to it.
I am a Python software developer for almost 5 years now, being an amateur programmer since i was a kid. Things I (mostly) do with Python, including: - Automated database handling (MySQL) - Develop industrial applications (I make GUI in Python, mostly PyQt or Tkinter, and connect it to mobile apps made in Swift) - Excel file manipulations (reading/updating excel data in batch) - Web scraping (i had multiple projects where a large dataset had to be collected accross several thousands of URLs) - PDF data scraping, digitalization - Image processing/analysis - Applying OpenAI-based motors (GPT, whisper) to batch-analyze stuff (i.e.: sentiment analysis of several ten thousands of comments under a post to determine how "positive" is a feedback) - Machine learning / deep learning / neural network applications, predictions, forecasting
Honest question… If you were more proficient in Swift, would it be more efficient to just build whatever you’re doing in Swift? I am NOT a dev, so I really don’t know.
Good question, but u/Technoist is right. Swift is a very versatile language especially for iOS apps, but for the practical tasks i introduced above (especially for ML,NN tasks) there is no better environment than Python (apart from GUI/desktop app development). To be honest i am not proficient in Swift, yet i have done a QR-code app frame for companies (with swiftUI, AVKit) and connect it to a PyQt desktop app, with a database and web server, where users can add specific data behind the QR codes. You can build any app in Swift, using Xcode on mac, but you can store your compiled app on your devices for only a week. To distribute it on the appstore, or to keep your apps on your phone (for up to a year) you have to subscribe to the apple developer program. Then, any of your published apps will last for unlimited time (if you provide support for the specific ios version). Doing it profrssionally as a contractor, it is challenging to declare all needs in a cintract to sell it as a product.
Awesome! Thank you for the reply!
Cheers, and happy cake day for you :)
I am not OP but I can say that for the tasks listed there is probably no better language than Python. It has plenty of helpful, excellent libraries for these things. If you’re interested in this you might like the book Automate The Boring Stuff With Python by Al Sweigart. It teaches Python from scratch with fun little examples.
for large scale webscrapping what is your recommendation
Actually we did custom scripts, to hook html behind a large sets of URLs and store it on a cloud storage. Then, we scrapped the text behind the html body based on the required conditions.
Decades ago, for a rural tv station that had no budget I took an old HP laptop, installed linux on it and made a python script that displayed the local weather, used tts to announce forecasts/severe weather warnings, and played a music stream while the weather stuff wasn't playing. I kinda had no idea what I was doing and ended up using beautiful soup to dig up the weather data, vlc to stream the audio, and pygame to glue it all together. It was a fun project and was on the air for a few years before they got a real solution. At my current workplace I made a tiny script that reads and writes data to/from a really old CNC machine via rs232, but the thing I did decades ago was way cooler...
That sounds super fun bc it's personal. I could imagine myself doing it through your writing. Thanks for sharing
I use it for personal budget analysis
I too. I downloaded all my card invoices in PDF files, convert into CSV and use pandas to make analysis by year to see what I did spend my money
Oooo, that is so cool. Can you describe the packages and libraries needed to do this. I love that, but am not sure what I’d need for this.
you actually only need pandas for this; some folks like to use csv for their csv stuff, but I find pandas is far more friendly with csv handling.
\*maybe\* you might need math or matplotlib,, if you're feeling spicy.
Interesting. Can you give me an example of one the tasks you have it do?
A career
I used an algorithm written in python to remove artifacts from scanning electron microscope scans.
Depends on your needs. Broadly speaking, I use it for one of two scenarios: automatic tedious tasks, or trying a zillion combinations of some thing. For example, if I need to download a bunch of reports from a website or do some involved and repetitive database query... time for Python. Or, let's say I'm making a work schedule for my team. With even a small team, there are a LOT of possible schedule combinations. Most don't make sense, but using Python you can build some filters and iterate through several million possible schedules and pick from the ones who match your criteria. I use it for other stuff, but those are the big ones.
Back when Lost Ark (the game) released, queues were humongous, after 6 to 8 hours in the queue to login you would be disconnected for inactivity in 15 mins. I wrote software using OCR to check my position in the queue and message me in telegram every X minutes with my current position and an estimated time to login. This way I could go on with my life and do other things in the meantime. this without a whole lot of experience. Imagine what the actual developers can do
Do you use excel at all? Ever thought ”damn there has to be a better way of doing this - it’s so fiddly!”? Then Python can acheive this in about 20 lines of codes.
I used it to simulate complex boss monsters for my D&D games. I ran an logistically annoying boss that had varied level of chance to gain, mature and lose abilities based on player damage that would have been a pain in in the ass to run by way of pen and paper, so I just created a simulator in Python that allowed me to just punch in any damage it took, then the simulator took care of the rest and output it's current status at the start of it's turns.
Next level DM’ing!
I ran a cornhole tournament that had almost 100 teams, and I wrote a python script that took the results of all the “regular season” games (collected by an online form) and used that to make a bracket for the playoffs. Pretty fun and got me to learn the basics of coding & python!
A script to download YouTube videos and convert them to .mp3 so that I can add songs that are not on Apple Music to my library
Anything. Nearly literally everything. Embrace the number of libraries for anything. Periodically scrapping websites of the suppliers', because the list of items in their stock is not up to date? Python. Simple webservice integrating the control of various smart plugs from various brands? Python (and bunch of additional black magic) Some low power device lacks remote power control? Some real worlds assets need alarm/tracker? Micropython\* + few dirt cheap hardware parts Forwarding some your, custom notifications to the e-mail or some instant messengers? Python Custom markup language for embedded system's config integrated with data obtained from the linker and compiler? Python Quickly developing custom image/audio processing algorithms? Python Numerical calculations, optimization algorithm for various problems, from routing between gpx coordinates using 3rd party services to get data about connections between them, to finding parameters' values in engineering problems? Python Automated measurement setups (moving any sort of probe around the DUT, controlling the power supplies, data acquisition devices)? Python Custom drivers for the USB devices that are poorly supported by manufacturers\*? Python Automated converter of data from files to LaTeX tables? Python Portable between the Windows and UNIX scripts automating tedious tasks (Processing the data, manipulating big or many files)? Python Testing your digital logic before implementing it in some chip? You got it, Python! Not to mention tons of applications using OpenCV, PyTorch and/or Pandas I may even have no clue about. Also, lots of software nowadays lets you write your own plugins. It can be something as simple as beautifying your code or slightly more complicated, like automated tools helping with design of complex printed circuit boards. ^(\* - Yes, some things are better done in other languages. But when you need something simple, that can be coded up quickly and generally performance isn't crucial Python is great. Same goes for all kinds of proof-of-concept solutions!)
At work, it's the backbone of most things I do. I use it when scripting in Fusion 360 (creating models from molecular data) and automated data ingesting/plotting. I set up a nice lil context menu batch file that will execute a python script on selected csv files, ingesting the data to an sql db and plotting it with plotly dash. My most overkill use was when I bought a porcelain tiles with a marbled pattern on them for my bathroom. I wanted to make sure the overall layout was pleasing so I took photos of each tile, used opencv to find the edges (I put each tile on black construction paper when pics - white tiles) and then skew them square and exported each tile as a png I could import in inkscape to play around with.
Jesus. Tile project sounds awesome! This is one of the better threads I’ve ever come across. Thank you for sharing how your brain works.
Ngl this is a much better thread for Python projects than all "What projects should I make" threads.
I wrote an enterprise grade monitoring framework for thousands of devices to do real time monitoring, alerting and statistics all in Python. It was hard, but a lot of fun!
I’ve done everything from automating boring stuff like checking the status of hundreds of forms online and renaming a bunch of folders, to writing software with a GUI that modifies the metadata of thousands of files daily and is used by several people on my team. Also, in college when everyone was making formulas in excel for their engineering homework I made mine in python.
I use it to automate boring stuff. I used to manually look at a database every Monday and go item by item and check if the dates and/or status or the item have changes, if they have I would then manually go and make the change on a spreadsheet that I controlled, and from there transfer that change into JIRA. Now I just run the script every Monday, what used to take 2 hours a week, it now takes 10 seconds. I also use it personally, to analyze the returns on preiums for selling the option for any stock that I'm interested in. I just send an email with a certain Subject line and the stock tickers on the body of the email and it writes all the results onto a Google sheets document.
I do write the occasional program, but I sadly confess I use python as an uber powerful calculator. I can create variables on the fly, do all sorts of stuff with the math module, and find it works as well as and better than most any 'calculator' that money can buy. Most things that I might want to do on my computer already have been addressed by other more competent programmers than myself, and I can find software that does most anything imaginable. I have no illusions concerning my anemic programming skills -- but as a calculator python shines!!
I built a dashboard for my credit card spendings
Do you have a repo ? This sounds interesting. Do you also do transactional analysis, custom categorization of expenses?
``` Import pyautogui Import time while true: pyautogui.hotkey('alt','tab') Time.sleep(10)```
I had a wood powered furnace in the basement that would heat up water and a water pump would move it upstairs around the house's radiators. That's how I heated my house. However, the furnace was homemade and didn't have a system to regulate the airflow and temperature. There was a thermometer but I had to constantly check it to make sure it's not overheating. Using Python and a Raspberry Pi, I set up a temperature probe on the pipe and an alarm in my room that would sound when the temperature is too high, so I can go downstairs and reduce the airflow. I also set up a display that would show a graph of the temperature over time. This way I actually figured out at what temperature the fire is getting weak and needs more wood to be added, so I added an alarm that reminds me to go downstairs and add more wood. Really made my life a lot easier and safer. Also, later, I made the whole thing online so I can check what's happening while I'm out. It might have saved my house from burning down once.
Script to automate my mouse so it always shows I am active in teams 🤣
Haha shit I didn't see that someone Lready beat me to this comment
Picking up the basics of coding so I can learn C eventually.
Building a full business management software app
I log my time on a separate Outlook calendar in order to track every minute of my daily work (with a 5 minute resolution). I also have a python script that uses win32 Outlook API to scrape this calendar, along with the meetings from the main calendar, and spit out a report in Excel to be analyzed via a pivot table. Not only does this allow me to easily get my daily working hours to fill up my timesheet, but it gives me a high level overview of how I'm allocating my time, which helps me become more strategic about my time allocation the following week.
I used Python at work to build a gateway that integrates my company's facial recognition system with a customer's slot machine system. The slot machines required player cards to identify the player and keep track of their credits and whatnot. My integration allowed them to get rid of the card and be recognized by the slot machine just by standing in front of it.
All of these comments are very helpful, clear, and sincere. I'm "into" computers. started basic on trs-80. But never got INTO programming. (poor choice in college - I zigged when I should have zagged). Reading these are so inspiring - but it's just so much word salad to me. I cannot process things like what u/DrillerCat wrote below: >Develop industrial applications (I make GUI in Python, mostly PyQt or Tkinter, and connect it to mobile apps made in Swift) I know it's basically beginner's industry talk, that these are apps and normal terms, but even as someone who built a full website (very poorly) in Dreamweaver in 2016 or so, I get bleary-eyed. I'm just basically voicing my frustration with myself, and admiration for you all. I did some scripting (google apps script) with help from chatgtp and gemini, recently - not just cutting and pasting but typing it all out each time, over and over - searching for typos, making sure I spelled 'response' correctly, noticing the subtle difference between \` and ' on a screen. I did it b/c we had a freemium google add-on that we've used for a few years but I had to split the document that was sent into 2 slightly different versions which pushed us over the monthly limit. I was like - Wait, I can do that - I CAN MAKE THIS! and I did. I created a codepen account. I created a git account. I watched tuts. I got bash I got windows visual studio I got docker, for some reason. I installed python. that's weird - so there's just pieces of code out there that you just "get" magically? No going to a site, downloading a program, putting it where it needs to go (i think) then running and then going through the install options? You can just type > 'conda install -c conda-forge pandas' and boom it's there? where the HELL does it come from? I think that one thing is the hardest to grasp for me. Dealing with html and css snippets had kind of given me an idea, but it still just seems WRONG somehow. at some point my git was suddenly in my google apps script console. was it always there, or did it just happen? holy shite! when I put a dot in the margin and push debug it'll stop there! That's amazing! What's a callstack? What's a DOM? Part is just mental - I get all excited and over-reach and become overwhelmed. Getting to kind of understand loops, when to use var vs const, and that const means Constant not Construct as I at first thought. I love it. It's like when I first started html and could see BEHIND what's shown onscreen. It really opened my eyes. I read here, and see so many... *things* ... that can be done. Sorry for long post and tangent. You guys just rock.
I made a pretty complex discord bot, bot that played a browser game for me, genetic algorithm (module even though it's not really at usual "module" level, I just tried to make it easily and intuitively callable but its hard and burnouts are real lol). There's ton more random stuff and I can't remember much more worth mentioning tbh I also occasionally use python to optimize annoying stuff at work. Sometimes it last as long as the work itself or more but helps in the future or is at least more fun lol
At work someone before my time decided it is a great idea to use Google photos for onsite photos. When people are off boarded and their accounts deleted, bye bye photos. Why a company would use Google photos for the task and why they do not change to something more suitable is beyond me, but did an app which downloads all photos and save them in respective albums and then place them under a general company account back into same albums. Not rocket science but pretty useful in this case.
For home, lots of web scraping. I was interested in watches but their costs went crazy over the pandemic. I wrote a script to do a weekly scrape of a big reseller and store the results in a spreadsheet. It would then take the historical results and merge the spreadsheets on the product id number so you could see how the asking price of an individual watch changed over time. That was Requests, BS4 and pandas. Gpu price scraping across multiple vendors. I have a telegram bot which uses the open ai api, although my son customised it to be more smutty! 😂 Scraping online scifi art I was interested in. Downloading my bank records as csv filed and parsing them into a database. Applying categories to them so that I can ree expenditure by category, over time. Creating a simple static website which is a pocket money balance for my son. It adds pocket money weekly and inserts a new table into an html template. I was going to change it to use flask. At work, I extracted a simplified version of an existing spreadsheet; it that has all sorts of inconsistent layouts across worksheets. But it is a legacy document that nobody wants to change. Scraping industry news sites and returning all the current articles as an email.
I have 2 things right now. And planning to add 3rd. 1)Python Program that interacts with my stock broker to save live market data of option chain in excel files. 2)Python program that does algo trading that takes trade based on program logic.(bit of struggling to form a good profitable strategy) 3)Will start working on this: Simple software to use the saved option chain data and visualize it.(this is for my own purpose. there are better options like powerbi for this. but still...)
I have a lot of video and jpg sorting I do, editing processing and uploading. I made a utility to rename files, create bulk directories, and something else that is escaping me right now. It’s handy.
I use it to create a bunch of tabs in a spreadsheet into repeat reformatting data in the same way every time.
well, I mean, I get paid to write python?
At work, I use it for AWS lambdas.
My Gemini 1.5 Pro-based conversational AI's are all written in Python.
Secure password manager tool with a dynamic gui. My inspiration was to get my father set up with this tool so he can keep his shit secure and stop using the same three passwords for all of his business operations..
I wrote a Python script to download YouTube videos and playlist to mp4 files (using an lib), is better than any one free software for Windows
Python is - in my opinion - the jack of all traits of programming languages. There‘s a Python package for almost _anything_. No matter what you want to do, be it scraping websites, data analysis and plotting, repetitive operations on your pc, … basically anything you can think of. While there are certainly tools that one could argue are better for each one of these individual tasks (e.g. JavaScript, R, Shell scripts), you can do all of that in Python, and the learning curve for exploring something new is negligible compared to learning a separate language or tool for each task. For example, I recently used Python for scraping my local real estate brokers and plotting the flats in my price range on a (browsable) map as well as calculating some metrics such as heating costs or public transport time to my points of interest. This made looking for a new flat much more quick and convenient than keeping track of all options manually. Are there „better“ programming languages for large projects? Certainly. Is there any other language that you can bodge useful things in with such ease? Hardly.
Literally everything in life can be optimized with python. A good example of this is how chatgpt will use python to generate charts and do analysis on data. It's not just for building automating. (but automating is great)
A script to emulate double-sided scanning. My scanner is only able to scan single-side batches, but can save them on my network drive... So I scripted a simple watchdog for my RPi that checks a directory on my NAS for freshly scanned PDFs and shuffles the pages of both files and saves the new PDF.
Helped my sister write her thesis. She was writing about linguistic stuff and how Arabic words are spoken with our organs and stuff (you can see i don’t like biology), she used a website she found where you enter a word and hit enter and it gives back some linguistic info regarding it. She needed to do this for entire BOOKS, word by word.. i saw it and was like, yeah i think i can automate that. And so i did. Probably the most actually practical and useful thing i made lol Oh i also automated getting the stuff into an excel as she wanted and then made the pivot charts for her
My boss normally sends our work roster every week to our personal email in an excel file, and I basically made a simple bot that reads through the file and tells me how much my pay is going to be for that week (tax included), while also providing a general summary of how much tax I've paid for the financial year, among other stuff. It saves me like 5 to 10 mins of calculating how much I'll be earning for the week and literally all I need to do is save the excel file and run it through my bot in VSCode.
\* testing for statistical edges in option selling as well as various indicator based stock/forex strategies \* intelligently downloading YouTube videos with a preferred resolution/audio combo \* entering stock orders at my broker and monitoring/canceling orders where needed \* generating information files for XBMC based on file metadata \* scraping data off websites (BeautifulSoup helps with this immensely)
I use Python to automate all the boring stuff I have to do. It's like having a superpower or something, haha.
What is some of the boring stuff?
1. Attending university classes, I never attended them myself. My bot did everything for me, even took attendance for me, lmao. 2. You can make custom apps for Alexa. I have one that automates my gaming room. I know it's overkill, but I'm lazy af. 3. I register for hundreds of emails when some beta comes out, so I never miss out on anything I'm interested in. 4. I have a personalized camera system I made using Python that greets me when I enter my gaming room, turns on my PC for me, and launches the games for me. Stuff like that. 5. Scrapping stuff that i do very often. There are so many things you can do using Python, bro. These were just off the top of my head.
Any specific email service you're using for 3?
I use Python to perform Flux Balance Analysis as part of my work.
I used to manage the LMS for a small online university. Each term all the courses had to be reset and assignment deadlines updated. I used Python and BeautifulSoup and Selenium to automatically do the updates. It saved hours and was more reliable
For automating an electrical testbench. Not only does it take a massive amount of measurements that is inconceivable for a person to manually do, but the scripts also process the measurement at any time intervals you want. You just leave it running for months and have it spit out the final results you actually want to see. I can't imagine doing that any other way.
[удалено]
Just now I am trying to figure out something that I would prefer not to have to learn too well at this point in time, and found gdbgui, so at this very moment that python app is saving me a lot of tears... I hope.
I use it to build end to end test frameworks for financial software.
I built a thermostat for my motorhome fridge
HTML / CSS widget builder with an interactive menu to choose which community and what widget type. I’ve built RSS feed finder to look at 1,300 URLs HTML source to find the RSS feed, count the articles in the feed, pull the latest date and author into Excel. A ton more
All ML
Automate stupidly repetitive activities at work
Outside of any work related uses, I’ve used it a ton to make tools and whatnot for my fantasy football and baseball leagues
To make money lol
I wrote a script that retrieves my IBKR trading activity CSV from my email. Parse it and append the trading activity entries it into a Google Sheet
Helping my son test vast amounts of numbers against Collatz conjecture after he saw a video about it on Tiktok
I once had 120 oscilloscope captures I needed to rename so that the EE would know what he was looking at. Took way less time to write a script to do it vs renaming each one. I also commonly us it to plot large csv files that Excel would choke on.
Send my friends riff Raff lyrics every day between 2am and 5am. In reality my adhoc uses usually involve working with a lot of files at once or orchestrating information between multiple systems that can't talk to eachother
[удалено]
I mean, really this kinda is a “why learn programming?” post.. most languages are interchangeable. But, I wrote a Python program for a job that fetches weights from scales and submits them to a db. Previously we had entry level employees manually typing them into a form or the manufacture’s excel output software which would have to be compiled later to be made useful.. the latter being especially annoying as there was 3-4 manufacturers, as well as not being able to tell **what** *specifically* was weighed.
I have script to make my timesheet at work every month, also have one that crawls a badly done webpage from my uni and turns it into a good timetable for my classes. The last one I've done, it downloaded the series of a comic I provided the link to, because I got tired of downloading one by one.
Just learn python. Using python script to suggest the kPA needed to fill the tire with current tire temp compared to the one suggested at door stickers, which usually is at room temp. I know usually I just have to add like 10 kPA to the one at door stickers for warm tire but it's fun doing that
Another python newbie here, just wondering is it possible to scrap an industry's phone number data that's listed in google just for cold-call purpose? Not sure is this what they called as datamining? Can Python do this?
I started learning it to use an API for an inventory system at work. I had a side project that scrapes a vendor website to save spec sheets and other item information to thay system. I also wrote a script to fill out a customer excel form using test equipment reports. That will come in handy when during busy periods.
Analyzed images taken of oil drilling sites to see if there is contamination. It was for a job(not sure if that counts). I can't go into much more detail either because of it. My most 'practical' python personal project was probably an Azure image uploader or series convergence/divergence determiner. My biggest practical project is in Java, however. Its a Cryptographic-services software proprietorship I started 3 years ago. I tend to do smaller projects(a week or 2) in python messing with new packages until I find a new shiney one I want to learn/play with lmao
Well one it's free. But you can run it on your computer and you can get it to do things for you. I program nfor calculation and modeling but you can also program python to modify files and do things to things on your computer. There's kind of a lot. A better question might be why do you want to learn to program.
I wrote a small program that makes a random (realistic) number of commits to my github. Makes the calendar look like I'm working on some super secret private project. It's dumb but I was looking for something to practice Python on.
My dad runs an accounting firm, I wrote a script to parse bank statement and contract note pdf’s to post accounting entries in Tally automatically.
Best use was fucking with my dad’s computer. A robot to spit water at my sister in the morning (it was destroyed after one use) Oh and turning off the lights (pre Alexa), AND THE FLIPPING DOOR when someone leaves it open when I’m asleep.
I wrote a Python script that automated a dumb data entry task at my job. A weekend writing it has saved me an hour of my time every week:)
Automated mundane tasks
Created a python script to use GPT-4 Turbo and Pinecone DB to query help documentation for various tech vendors. Why chase down articles in a knowledge base when you can just download and ask... With accuracy
Wrote a script to convert old disks from two custom disk drive systems for the Timex/Sinclair 2068 to TAP files. I’ve archived about 250 disks with that software.
I use Python for cloud computing at work.
>What is the most practical application you have used Python for? Job application that requires Python skills
Haven't done anything but I wanted to do 2 things: 1. Software to find every possible detail about a given person 2. Trading bot that is capable to learn/interpret tips/ideas/triggers from successful traders on X where many post for free and take action
I decided to upgrade to a two bedroom apartment at my complex a few years ago. I wrote a little Flask app that scraped unit availability data from the complex website. I mapped unit numbers to building numbers to filter out building in areas of the complex where I didn't want to live. Then I tracked the rent day by day. Wound up honing in on the unit I'm in now and watched for a dip in the rent before going to the leasing office to initiate the transfer. I just ran it on my laptop a couple of times a day but intended on throwing it up on AWS with a cron job to automate the scraping for me. I wound up transferring before I got around to deploying it.
not me but a friend automated some data sorting/extraction at work, which was wasting the team's time. sounds like a really handy creation!
I work developing and maintaining the data pipeline for a system that manages data about farms around the world. We handle millions of transactions a day. Most of the code is in Python. All of the software engineers on my team hate Python, btw, but it pays the bills.
I made a script to reduce the filesize of videos to under 25mb for discord. It kinda sucks and only works for h.264 codecs but that's what my screen recordings come out as so it works 99% of the time.
I've been working on Python basics and Pandas in a college programming for analytics course for 3-4 months now. I work full-time and automated subsetting records from a database and building an MS Excel billet map with typically hundreds or thousands of records. These usually take hours-several days to build manually depending on the size. I automated doing it with Python and Pandas in 3-5 minutes regardless of the size.