Python 3 limits


I had decided that I will port my code from Python 2 to Python 3 and with the help of PyCharm the refactoring that I wanted to do was going completely smoothly. I decided in the middle to try something new. I thought when I was restarting Python then maybe try something new.

While doing this I hit Python 3 limits again. The library support is still not as good as for Python 2.  The first problem that I faced was with mechanize library. That was when I was trying to get some automation scripts done for logging into websites as per a schedule. Everyone wastes a lot of time online and I wanted to control it. Had the schedule but just opening websites by webbrowser module was not good enough. Why not use Python to login also? Mechanize was just what was needed. But it didn’t support Python 3. Shit. I found a solution to that. Using Selenium I did a test run and was able to login into http://projecteuler.net/ using my username and password. That I can work with now. But still mechanize might have been easier to work with.

Just 2 days back I had the idea of trying a bit of AI. Maybe a small bot or something? Natural Language processing for intelligent answers maybe? Maybe some simple  image recognition. Just the flight of thoughts that come and go every now and then. These are not like anything that I have tried before but what’s the problem with trying? I decided to go with Natural Language processing. Shit again. NLTK – the library needed for this had support for Python 2 only.

I was left thinking whether the advice that I got on Python mailing list for using Python 3 for new software actually good? Why actually did I change to Python 3? UNICODE is not something that I have even messed with once in Python. So why? I installed Python 2 but pip was missing. That’s a good thing that they did in Python 3,4, not that it actually helps with half of things not supporting Py3.4 yet.

Anyways I just noted down the ideas that came and decided to thing a bit more before going further. Hope someone reads it and decides to not go with “newest and latest” unless they have the time to port libraries as well.

Trying to learn Haskell and feeling weird


I had started Python with the main aim of breaking into functional programming. Had read someplace wrong that Python is functional programming language. The time spent on learning it and using it was great. I didn’t get what I started out for but learned the ease of using a high level language and a dynamic one. Coming from mainly C background that changed my perspective a lot about programming.

I decided to try it once more, functional programming. This time I am going with Haskell which is said to be the ultimate functional programming language. Let’s see how far I can get. For the 1-2 days that I have actually tried using Haskell it has been a weird experience. In a good way. It’s different somehow. Couldn’t place how.

Placed project C Programming Made a Game on github


I placed the project for which I had made my old blog on github. I had the versions backed up but not via git. Back then I didn’t know about versioning software like git and mercurial. I was cleaning up my codebase and thought that converting the code dumps of previous versions of the software into git would make it much easier to manage the whole thing if I needed.

I also added the C++ programs’s source code which I had used for the project into the github repository. They had helped me back then take care of repetitive tasks a lot. That was kinda the project where I understood the real value of automating tasks and one of the reasons I started using Python.

I don’t think I am going to update it anymore but good to have that on github. It gets backed up. I am thinking about adding the setup files that I have for various versions on github or not. Still not decided on that.

If I ever thought of continuing the project I’ll do that from scratch in Python/PySide which I have been learning for some time.

If anyone’s interested in seeing what the project was just have a look at the old blog.

Automating tasks in Python – Mass renaming of folders present in zip


So Python is getting useful to me for avoiding manual repetitive tasks. I used it to rename a lot of folders. There were around 700 folders in 28 zip files which I needed to rename and re-zip. That would have been really cumbersome if not for Python.

The folders’ name were like  0001 – abc 1 . Only that the numbers didn’t match. The first number was used for arrangement and it was the second number according to which I had to change the first number. Doing that for 700 folders would have been a real pain. Luckily I knew Python and wrote a script for doing this.
Here’s the Python script that I used.

import zipfile
import os
import shutil

CUR_DIR = os.getcwd()
FILES_IN_CUR_DIR = os.listdir(CUR_DIR)

def get_new_name(original_name):
    num = 0
    started= False
    for c in original_name[7:]:
        if not c.isdigit():
            if started is True:
                break
            continue
        else:
            started = True
            num *= 10
            num += int(c)
    return str(num).zfill(4) + original_name[4:]

def extract_files(zip_f, cur_path):
    with zipfile.ZipFile(zip_f, 'r') as zip_file:
        zip_file.extractall(cur_path)

def rename_files(cur_path):
    for files in os.listdir(cur_path):
        new_name = get_new_name(files)
        os.rename(os.path.join(cur_path, files),
                  os.path.join(cur_path, new_name))

def main():    
    files = [f for f in FILES_IN_CUR_DIR
             if f.endswith('.zip')]

    for f in files:
        cur_path = os.path.join(CUR_DIR, f[:-4])
        print('Extracting ' + f)
        extract_files(f, cur_path)
        print('Started renaming')
        rename_files(cur_path)
        print('Removing zip file ', f)
        os.remove(os.path.join(CUR_DIR, f))
        print('Writing back to ' + f)
        shutil.make_archive(cur_path, 'zip', cur_path)
        print('\n')if __name__ == "__main__":
    main()

This is just the first version that I used to get it to work. It creates temporary folders that I needed to delete but that was just one select and delete. Not a big thing. I plan to take care of the temporary folders too. The new one you can find in my github -> general -> Python33 -> Utilities maybe in a few days.

Python script for getting stats of a website


I have been interested for some time in a particular website – codereview.SE. It’s a website for getting your codes reviewed. I have been learning a lot from it. It’s still a test website so I wanted to find out whether it is growing or not. For that I decided to write a Python script for getting the stats on its front page and using them over many days to plot graphs.

Here’s the Python script

#! python3
import urllib.request
import datetime

FILE_NAME = 'data_file.txt'
CURRENT_URL = 'http://codereview.stackexchange.com/'

def today_date():
    return datetime.date.today().strftime('%d-%m-%Y')

def already_written():
    with open(FILE_NAME, 'a+') as f:
        f.seek(0)
        first_line = f.readline()
        if today_date() == first_line[:-1]:
            return True
        return False

def parse(line):
    """This separates the stat-name and associated number"""
    temp = [0, '']
    braces = False
    for c in line:
        if c == '<':
            braces = True
        elif c == '>':
            braces = False
        elif braces is True or c in [' ', ',', '%']:
            continue
        elif c.isdigit():
            temp[0] *= 10
            temp[0] += int(c)
        else:
            temp[1] += c

    return temp

def write_stats():
    '''This writes the stats into the file'''
    with open(FILE_NAME, 'r') as f:
        data = f.readlines()

    with open(FILE_NAME, 'w') as f:
        url_handle = urllib.request.urlopen(CURRENT_URL)

        f.write(today_date() + '\n')
        f.writelines(data[1:])
        f.write(today_date() + ',')

        for line in url_handle:
            temp_line = str(line)[2:-5]
            if 'stats-value' in temp_line and 'label' in temp_line:
                temp = parse(temp_line)
                f.write(str(temp[0]) + ',')

        f.write('\n')

def main():
    if not already_written():
        write_stats()

if __name__ == "__main__":
    main()

This will store the five stats shown on the front page of the website into a text file data_file in the following format

23-08-2013
dd-mm-yyyy, questions, answers, %answered, users, visitors/day

22-08-2013,9079,15335,88,26119,7407,
23-08-2013,9094,15354,88,26167,7585,

The above is the data I have currently in my file. The first line is for checking whether data has been added today or not. That will take care of duplicates in the file. The 2nd line was manually added to make sure I don’t forget what was what. Actually it was added so that I can use them as titles of graphs to be made for finding the growth of the website.

I have found that matplotlib library is really good for producing graphs. I’ll be using that for finding the growth. Obviously after I have some more data.

Making hierarchy of folders using Python


Suppose you want to make many folders aka directories for some purposes. Maybe you need to make many folders in a relative’s computers? Maybe you want to make many folders with same pattern in the names? Maybe someone has a habit of deleting files or formatting his hard drive and you have the responsibility of making a basic folder hierarchy>

I posted how to delete files so I thought to add one to make files also.

The hierarchy of directories that I am making is like this. Every indent means a subdirectory of the previous unindented one.

./project_euler
    ./001_050
    ./051_100
    ...
    ./401_450
./codechef
    ./easy
    ./medium
    ./hard
./spoj
./functions
./utilities

I made this script to learn Python mainly but it can be used to do same task of creating directories again and again easily. There are alternatives in Unix/Linux but in Windows it might be the easiest way.

#! python3
import os

TOP_LEVEL = ('spoj', 'functions', 'utilities', '_testing')
EULER_HIGHEST = 450
CODECHEF_FOL = ('easy', 'medium', 'hard')

def safe_make_folder(i):
    '''Makes a folder and its parent if not present'''
    try:
        os.makedirs(i)
    except:
        pass

def make_top_level():
    '''Makes folders with no subdirectories'''
    for i in TOP_LEVEL:
        safe_make_folder(i)

def make_euler_folders():
    '''Makes euler and its subdirectories'''
    for j in (os.path.join('project_euler', '{:03}_{:03}'.format(i, i + 49))
              for i in range(1,EULER_HIGHEST, 50)):
        safe_make_folder(j)

def make_codechef_folders():
    '''Makes codechef and its subdirectories'''
    for i in CODECHEF_FOL:
        safe_make_folder(os.path.join('codechef', i))

def main():
    make_top_level()
    make_euler_folders()
    make_codechef_folders()

if __name__ == "__main__":
    main()

This script’s comments are self-explanatory. Any questions? Just ask in the comments?