Python: Geeking my way to hotel points

Note: At the time of writing Egencia’s robots.txt (which says what a an automated process can access permits this use case)

 

The Challenge

I currently work for one of the top 4 IT consultancies as a Data Scientist and as a northerner that means lots of travelling to London… On the plus side it does mean I get to collect my fair share of hotel points, which brings me onto the reason for this blog.

I was sat in my hotel room one cold, dark and stormy night (just kidding I was down south) and I wondered if I could get notified when double points offers were available for my travel plans. This would massively boost my points earning capability! I thought this is going to be a piece of cake! So, I got on my laptop and went straight into the Python terminal to start tinkering. *typing* “import requests” “r = requests.get(‘www.the-website.com’)”… *stops typing*

Alas, I was thwarted by a login screen on the corporate travel booking website Egencia (the corporate version of Expedia) not only that but it required a session token and a myriad of other security features. The weeks went by with a few mentions of my project to my colleagues on how to get into this garrison of hotel offers. Then I read a great article on Medium about the Selenium web driver, using which you can mimic a human by programming the series of events that you would need to click or type to be able to access the information you require. In this article William Koehrsen shows how he used Selenium to upload his homework programmatically using python’s selenium package.

I finally had the key to the holy grail of points! Initially getting Selenium up and running wasn’t the easiest of tasks, mainly because I wanted it to be in its own Docker container and my notebook to be running in another container. In the end I gave up and used a python virtual environment on my laptop and a pre-cooked Selenium Docker container.

Now I could get to work, I started to inspect Egencia’s web page, found the fields needed for the login and I was in! I felt like James Bond, now all I had to do was navigate my way through a minefield of pop ups, hidden DOM objects, ‘load more’ buttons, etc… *sigh*, this solution worked but it had many problems, speed being a major one. However, I’ve still put the code up on GitHub for reference [link].

I was not happy with this solution and I really didnt feel like a good developer as I was relying on the View from the MVC framework which will enevitiably change a lot more frequently than the Model. So I put my developer hat on with one of my good friends Sia Gholami and went back to requests but this time created a session, the first thing I needed was to get all the different tokens from the page so I used ‘requests.get()’ inside the session, now since the session is still active if I now use ‘requests.post()’ I will have the same session tokens that I retrieved from the get request.

Authenticating with requests

Check out the repo

What now?

Now we just need to add a method for querying Egencia for a certain hotel brand in a specific city and for some set period of time, this is now just a case of creating a url string (I found this url by using the Chrome developer tools and inspecting the network traffic to see what happened when I tried booking my hotels manually)

Also note that there are some id’s we have to find first, namely the user id and the company id. Again I found out where I could get these from by following the network traffic using Chrome developer tools.

Check out the repo

 The final solution

Now I wanted to wrap this up nice and neatly, as I mentioned at the beginning I wanted to be notified and for this to be ran automatically so I looked to two other technologies, one more recent than the other. The age old Cron job for running jobs at a set time and Telegram which is a great free messaging application and you can make bots!

I whacked the class I made above into a separate file called egencia_obj.py then created a few helper functions. I also made a CSV (data.csv) using excel showing the following:

DateCityMax cost
02/04/2018Croydon85
03/04/2018Croydon85
04/04/2018Croydon85
05/04/2018Croydon85
06/04/2018Croydon85
07/04/2018
08/04/2018
09/04/2018Croydon85
10/04/2018Croydon85
11/04/2018Croydon85
12/04/2018Croydon85
13/04/2018Croydon85
14/04/2018
15/04/2018

Now when the script below runs it will message me with any Hilton offers it has gotten from Egencia’s website! Note I put all of my credentials into a separate file (settings.py)

Check out the repo

After this runs I get sent a message with all of the offers it finds! If I see anything I like I can then log in to Egencia and book it, I haven’t gotten it quite so far as having a ‘book it’ link but that could be my next stage for this project.

Heres a few request challenges for you

  • Automatically ‘log in’ to Netflix or Amazon Video and find your most recently watched film/series and make your desktop background the artwork image.
  • Do the same as I have but with Hilton, Marriot, Holiday Inn, whatever floats your boat.
  • If you’re feeling risky and I don’t advise this challenge, you could log into your online banking (depending on how their system works) then you could pull your statements and train a classifier to say whether a payment was on expenses or was a personal payment. This is something I’m waiting for with the open banking initiative, so the banks provide an API which would be more secure than storing plaintext passwords on your machine!

1 Comment

Leave a comment

Your email address will not be published. Required fields are marked *