Abstract Image for end of scraper project

Lambda web scraper project (week 8)

What Happened Last Week

In this final week, I was hoping to move my AWS Lamba functions over to Google Cloud Platform (GCP). Unfortunately, at least for me at this time it didn't work, the GCP Python 3.7 functions offering is currently in BETA which means it's available for testing but not general release.

I didn't just give GCP a try though, I also registered on a new platform https://binaris.com. Who suggest their offering provides functions that invoke in milliseconds and can replace any container or instance based service. Her also I unfortunately also got stuck, but here it was more the lack of documentation. There service is also not fully finished.

Are Functions For Me?

For the moment I say hell YES! It currently costs $0 for me to run them.

While I don't like the fact of currently being tied into AWS and not being able to easily switch to another platform, I haven't had to pay a dime for running my Lambda functions. 

No Amount Due

My monthly bills during development show:

  • November $0 for 1.250 Lambda-GB-Second
  • December $0 for 46.375 Lambda-GB-Second
  • January so far $0 for 129.700 Lambda-GB-Second

I estimate now that I am running my crawler every 2 hours on two pages by the end of January it will be about 600 Lambda-GB-Seconds. On AWS Lambda you currently get 400,000 GB-Seconds for free. 

You need to be wary if you think you'll become a high volume user. The service costs could change at any time and depending on your architecture it might be expensive to get away from their pricing model.

Videos from the project

As part of this 8-week project, I managed to upload over 2.5 hours of video to YouTube and talked extensively about GitLab CI/CD, Python2 to 3 migration, Unit testing and learning Python as well as writing AWS cloud formation templates.


YouTube playlist for the python web scraper project

What's Next

I'm not sure. Last week I did start playing with porting my solution to Docker, but I know running containers will certainly cost money. I need to sleep on it and will vlog about it next Friday here https://www.youtube.com/neilspink