CS50's Web Programming with Python and JavaScript

I recently audited Harvard's online class, CS50's Web Programming with Python and JavaScript.
It was a 12-lecture course covering:

• Git 
• HTML, CSS 
• Flask 
• SQL 
• ORMs, APIs 
• JavaScript 
• Front Ends 
• Django 
• Testing, 
• CI/CD 
• GitHub, Travis CI 
• Scalability 
• Security

I posted three new repositories on my GitHub from assignments from the class. 

Book review Flask App

GitHub

Website where users can log in, search for a book, go to a book's page, see rating data from GoodReads, and add a review. 

Web Socket Chat App

GitHub

This is another Flask website. It's a chat room website. It uses flask_socketio, which is an Ajax library for flask. Users can send and receive messages and change chat rooms, all while staying on the same web page. 

Pizza Order App

GitHub
Users create an account and log in. The order page allows the user to add pizzas to an order, customizing the toppings, crust, and size. Staff can add their pizzas and track pending, unfulfilled orders via Django's Administration page.

Reading digital well log files with lasio

A well log is a measurement with respect to depth or time of various physical characteristics in or around a well bore. Well logs are used extensively by the oil and gas industry. The universal digital format of well logs is the Log ASCII Standard (LAS).

Reading and interpreting well logs has been a common task for me both in college and at work. I have a project in mind for an app to store, view, interpret, and share well logs in a web app, so I want to be more familiar with the file format.

This Notebook is my first attempt at working with well logs using Python. I'm using lasio as my LAS parser.

My goals with the project were to

• Test if lasio works consistently with hundreds of well log files.
• Analyze the usage frequency of well logging tools and how standardized the syntax is between well logging tools in terms of how the tools are named and described.
• Plot some of the gamma ray, resistivity logs.

No test of a LAS parser would be complete without plotting the log data. Images of plotted well logs are quite large. It's common to have more than one data point per foot depth for each tool and a well can be many thousands of feet deep. Below is a cropped example of a plotted well log. Full logs can be found in the git repo. 

Cropped plot of resistivity and gamma ray logs

Forum Website

I made my first web application! 

It's a forum website. Think, reddit, or a site where people post and comment on things. I chose a forum, because some of my ideas down the road involve a website where users collaborate.

I built the project using Django, a Python web framework. Django allows for fast development and is built using a "don't repeat yourself" philosophy which minimizes redundant code.

After going through the official Django tutorial and poking around, I wanted to make something by myself.

Let's see it

Here's the Github repository. To run it locally from Command Prompt or Anaconda, navigate to the manage.py file and enter "python manage.py runserver". 

If you don't want to do that, I do have a few screenshots.

Index Page

The index page will look familiar to a reddit user. The eye grabbing content is the title of each post. In smaller text is when the post was submitted, who posted it, and where they posted it.
In the dark blue header, is a quick link to some of the website's forums. A list of all the website's forums is also available.

Forum page

Each Forum's page lists the forum's topics sorted to show the most recent. The forum's description is in the sidebar. The URL to get to a forum's page is really easy to read. It's just /f/[forum_name].

Topic page

Each topic's page contains the topic's body, followed by posts from other users. The topic page inherits the sidebar from its forum.

Final Thoughts

I used Visual Studio 2017 (VS) as my IDE. I never used VS for development before, but I found it very useful for web development. I'll definitely use it again for my next Django project.

I was happy that the front end web development class I took didn't go to waste. Bootstrap really adds a lot. 

Cross Over Stock Trading Algorithm

Sometimes I wonder if or when I should buy a stock, and I'll get a stock trading algorithm in my head. Would it beat holding the stock? How risky is it?

One way to access that is testing the algorithm on past market data. Here, I do just that.

Stock Market Analysis

• Use API to pull stock data
• Methods to analyze stock data
• Crossover trading concept 
• Backtest crossover trading strategy

Crossover algorithm trading periods

While this Notebook follows another blog post, I made significant changes to the code and added better visualizations.

Stock Trading Algorithm Testing

• Test the Crossover trading algorithm on all the stocks in the S&P 500
• Analyze results 

Return ratio of holding stock vs algorithm for S&P 500 stocks 

Crossover algorithm often exceeding hold when hold results in loss

GitHub for this project

Disclaimer: This is in no way financial advice. I am not a financial adviser.  This post and code were done as a data analysis exercise only.

Google's Deep Dream

Google's Deep Dream enhances patterns in an image similar to the way your mind perceives a face in the moon, an algorithmic pareidolia. I recently found a Jupyter Notebook, published by Google, showing how to produce the Deep Dream visuals. I thought I'd give it a shot.

If you have never heard of Deep Dream, this GIF shows its effect on an image.

This GIF shows how the number of Deep Dream iterations affects the image.

GitHub is not able to show two of the GIFs I created in my notebook due to file size limitations. I included them in this blog post (one I converted to video).

This video was made by applying the Deep Dream filter, zooming in, and looping that 200 times. There are some noticeable differences between this video and the original GIF. 

Here's how I used the Deep Dream program. 

I'm happy with how it turned out. I was able to see the capabilities of Deep Dream by iterating through various settings. I was able to use Python to turn the generated images into labeled GIFs and tiled images. I think it turned out to be an effective way to display the data.

Installation

This Notebook uses Python Imaging Library (PIL) which has yet to release a version for Python 3.X, so I'll need to run the notebook on Python 2.7. Good thing Anaconda makes managing different Python environments easy. I wonder why Google didn't use Pillow instead and run the project on Python 3.

Installing Caffe was more difficult than I expected. Some issues I came across are:

• Instead of using a package manager, Caffe needs to be built with Visual Studio.
• There are multiple versions of Caffe for Windows available on GitHub. The two I tried were from Berkeley Vision and Learning Center (BVLC) and Microsoft. 
• When I tried to build Microsoft's Caffe, the build failed. This was due to the build requiring a debug version of Python. Oddly the installation instructions made no mention of this requirement. It's only mentioned in the .PROPS file where the build's variables are set. 
• I was able to build the BVLC version of Caffe, even though it mentioned the same debug requirement in the .PROPS file. 

The Microsoft Surface I'm running this on has an integrated GPU, which does not support NVIDIA's CUDA. I had to use a CpuBuildOnly build. At default settings it took about 5 minutes to render an image using most filters. I would look into a GPU build for a larger project.

GitHub link for this project

For more Deep Dreams, see this Instagram.

What's in this folder?

Are you ever unsure what files are on in a folder on your computer or network drive? Small folders are easy to click and see, but large folders with many subfolders can be too much to look through.

Using Python, I made a script to walk through a file directory and output the data in an easy to read tabular format.

Here's the project on GitHub.

Tkinter GUI

I added a basic GUI using Tkinter. Tkinter has a file dialogue that works for selecting a folder. The user has the option to specify how far down the directory to see. The user also can toggle if file names in the folders will be included in the output.

Tkinter GUI

File dialogue GUI

Output

The output includes the path, folder/file name, folder/file size, count of files and folders both in that folder and subfolders, and a list of file types sorted by occurrence. A list of files in the folders can also be added to the list. The output is saved as a CSV file. The output file name is based on the name of the directory, so the script can be ran for multiple directories without overwriting other searches.

Example Output

Numpy on NOAA Weather Data in IPython Notebook

I've been meaning to use Python more for data analysis. I took a Lynda class focusing on Matplotlib, Numpy and Pandas. This post focuses on the exercise analyzing NOAA weather data using Numpy and Matplotlib. Here is the IPython Notebook on GitHub. GitHub will allow you to view the notebook as an HTML document.