Fore­words

A while back I decided to try and learn Python for the hell of it as it seems like an inter­est­ing lan­guage and has some of the most con­cise and user-friendly syn­tax. Hav­ing spent some time going through a num­ber of dif­fer­ent learn­ing sources and mate­ri­als (like the offi­cial site python.org which has a very help­ful tuto­r­ial sec­tion) I have put together a set of notes I made as I was learn­ing and hope­fully they can be use­ful to you as a quick list of how-to code snippets.

All the code snap­shots I’m show­ing here are taken from the IDLE Python shell.

Lists

To cre­ate a new list:

clip_image001

 

Lists are NOT immutable:

clip_image002

 

Use the in key­word to check whether an ele­ment is in the spec­i­fied list:

clip_image003

 

Nest­ing lists:

clip_image004

 

The min and max func­tions:

clip_image005

 

The list func­tion – you can use the list() func­tion to con­vert a tuple to a list:

clip_image006

 

Ele­ment val­ues of a tuple can­not be changed and tuple ele­ments are put between paren­the­sis instead of square bracket:

clip_image007

 

Delet­ing an item from list:

clip_image008

or you can use the remove() function:

clip_image009

 

Replace por­tion of list with slicing:

clip_image010

 

Insert a list into another list with slicing:

clip_image011

 

Delete a por­tion of list with slicing:

clip_image012

 

Append­ing to a list by using sim­ple concatenation:

clip_image013

or use append or extend, the dif­fer­ence being append adds a sin­gle ele­ment to the list where as extend works like the con­cate­na­tion above.

clip_image014

now com­pare this to extend:

clip_image015

 

Like in Javascript, you can use a list like a stack (FILO) too:

clip_image016

 

You can also use a list as a queue (FIFO) using the collections.deque function:

clip_image017

 

Sort­ing a list:

clip_image018

you can do the same to a string too using the sorted function:

clip_image019

 

To con­struct an empty tuple:

clip_image020

To con­struct a tuple with a sin­gle item:

clip_image021

 

You can unpack a tuple or list (like the pat­tern match­ing in F#):

clip_image022

clip_image023

 

There must be the same num­ber of ele­ments on the left as the tuple on the right:

clip_image024

 

Use the range() func­tion to gen­er­ate a range of integers:

clip_image025

 

Use the fil­ter() func­tion to fil­ter a list:

clip_image026

 

Use the map() func­tion to project a sequence’s items to some­thing else:

clip_image027

you can also use it like the zip() method in F# by pass­ing in mul­ti­ple sequences:

clip_image028

if the lists are not of equal length, None is used to fill in the gap:

clip_image029

 

Use the reduce() func­tion to return a sin­gle value from a list of ele­ment, e.g. to sum the num­bers 1–4:

clip_image030

you can also pass in a third argu­ment to indi­cate the start­ing value of the accumulator:

clip_image031

 

You can remove an item from a list using its index with the del statement:

clip_image032

note that del state­ment doesn’t return any values.

You can also use it to delete the entire list or part of the list:

clip_image033

or to delete the vari­able itself:

clip_image034

 

List com­pre­hen­sions (sim­i­lar to those in F#):

clip_image035

 

If the result is a tuple, then it must be parenthesized:

clip_image036

 

You can add addi­tional filters:

clip_image037

 

Or you can have a loop inside another loop:

clip_image038

 

Nested List Com­pre­hen­sions, e.g. to turn the columns of a matrix into rows:

clip_image039

remem­ber, read nested com­pre­hen­sions from right to left!

Nested com­pre­hen­sions is a pow­er­ful tool but adds com­plex­ity, where pos­si­ble, use built-in func­tions. E.g. the above can be done using zip():

clip_image040

 

When loop­ing through a sequence, the posi­tion index and cor­re­spond­ing value can be retrieved at the same time using the enu­mer­ate() function:

clip_image041

 

You can also use zip() func­tion to loop over two or more sequences at the same time:

clip_image042

Share

Fore­words

A while back I decided to try and learn Python for the hell of it as it seems like an inter­est­ing lan­guage and has some of the most con­cise and user-friendly syn­tax. Hav­ing spent some time going through a num­ber of dif­fer­ent learn­ing sources and mate­ri­als (like the offi­cial site python.org which has a very help­ful tuto­r­ial sec­tion) I have put together a set of notes I made as I was learn­ing and hope­fully they can be use­ful to you as a quick list of how-to code snippets.

All the code snap­shots I’m show­ing here are taken from the IDLE Python shell.

Basics

Com­ments:

clip_image001[10]

 

Vari­able assignment:

clip_image002[10]

 

Arith­metic:

clip_image003[10]

 

Power:

clip_image004[8]

 

Absolute value:

clip_image005[8]

 

Get­ting user input:

clip_image006[8]

 

raw_input vs input:

raw_input always con­tains string, input can con­tain any object, even a calculation:

clip_image007[8]

 

Import mod­ules:

clip_image008[8]

 

Func­tions as first class objects:

clip_image009[8]

 

If-elseif-else:

clip_image028[8]

 

The is oper­a­tor checks if two vari­ables refers to the SAME object:

clip_image029[8]

on the other hand:

clip_image030[8]

The is not oper­a­tor does the reverse.

 

The and and or log­i­cal oper­a­tors, same as && and || in C# respec­tively. You can use the not oper­a­tor to negate the out­come of a boolean comparison.

 

You can chain com­par­isons, e.g. is the value of x greater than or equal to 5 and less than or equal to 10?

clip_image031[8]

 

You may com­pare sequence objects of the same type, which uses lex­i­co­graph­i­cal order­ing — com­pare the first two, and if they dif­fer then that’s the out­come of the com­par­i­son, else com­pare the next two, and so on:

clip_image032[8]

 

Strings

Strings can use dou­ble or sin­gle quotes interchangeably:

clip_image010[8]

Escape char­ac­ter:

clip_image011[8]

Span­ning across mul­ti­ple lines – a back­slash (\) as the last char­ac­ter on the line indi­cates that the next time is a log­i­cal con­tin­u­a­tion of this line:

clip_image012[8]

or you can sur­round them in a pair of match­ing triple quotes: “”” or ”’:

clip_image013[8]

 

String con­ver­sion using the str() function:

clip_image014[8]

The repr func­tion – the repr func­tion returns a canon­i­cal string rep­re­sen­ta­tion of the object, back-ticks () do the same thing (they are sim­i­lar to the ToString() method on C#‘s objects:

clip_image015[8]

 

String con­cate­na­tion:

clip_image016[8]

 

Slic­ing a string:

clip_image017[8]

You can also use neg­a­tive index, in which case it starts count­ing from the right:

clip_image018[8]

note: message[0] = message[-0], see how the indices are mapped:

clip_image019[8]

you can also set up steps in the slicing:

clip_image020[8]

sim­i­larly to before, you can slice back­wards too:

clip_image021[8]

 

Get length of string:

clip_image022[8]

 

Strings are IMMUTABLE!

 

For­mat­ting strings:

clip_image023[8]

 

Find­ing sub­string (returns the index of the start of the first match):

clip_image024[8]

 

Join­ing strings:

clip_image025[8]

 

Chang­ing the case of strings:

clip_image026[8]

 

Replac­ing por­tions of a string:

clip_image027[8]

Share

I’ve cov­ered the topic of using Smart­Thread­Pool and the frame­work thread pool in more details here and here, this post will instead focus on a more spe­cific sce­nario where the rate of new work items being queued out­strips the pool’s abil­ity to process those items and what hap­pens then.

First, let’s try to quan­tify the work items being queued when you do some­thing like this:

   1: var threadPool = new SmartThreadPool();

   2: var result = threadPool.QueueWorkItem(....);

The work item being queued is a del­e­gate of some sort, basi­cally some piece of code that needs to be run, until a thread in the pool becomes avail­able and process the work item, it’ll sim­ply stay in mem­ory as a bunch of 1’s and 0’s just like every­thing else.

Now, if new work items are queued at a faster rate than the threads in the pool are able to process them, it’s easy to imag­ine that the amount of mem­ory required to keep the del­e­gates will fol­low an upward trend until you even­tu­ally run out of avail­able mem­ory and an Out­OfMem­o­ryEx­cep­tion gets thrown.

Does that sound like a rea­son­able assump­tion? So let’s find out what actu­ally happens!

Test 1 – Sim­ple delegate

To sim­u­late a sce­nario where the thread pool gets over­run by work items, I’m going to instan­ti­ate a new smart thread pool and make sure there’s only one thread in the pool at all times. Then I recur­sively queue up an action which puts the thread (the one in the pool) to sleep for a long time so that there’s no threads to process sub­se­quent work items:

   1: // instantiate a basic smt with only one thread in the pool

   2: var threadpool = new SmartThreadPool(new STPStartInfo

   3:                                          {

   4:                                              MaxWorkerThreads = 1,

   5:                                              MinWorkerThreads = 1,

   6:                                          });

   7:  

   8: var queuedItemCount = 0;

   9: try

  10: {

  11:     // keep queuing a new items which just put the one and only thread

  12:     // in the threadpool to sleep for a very long time

  13:     while (true)

  14:     {

  15:         // put the thread to sleep for a long long time so it can't handle anymore

  16:         // queued work items

  17:         threadpool.QueueWorkItem(() => Thread.Sleep(10000000));

  18:         queuedItemCount++;

  19:     }

  20: }

  21: catch (OutOfMemoryException)

  22: {

  23:     Console.WriteLine("OutOfMemoryException caught after queuing {0} work items", queuedItemCount);

  24: }

The result? As expected, the mem­ory used by the process went on a pretty steep climb and within a minute it bombed out after eat­ing up just over 1.8GB of RAM:

image 

image

All the while we man­aged to queue up 7205254 instances of the sim­ple del­e­gate used in this test, keep this num­ber in mind as we look at what hap­pens when the clo­sure also requires some expen­sive piece of data to be kept around in mem­ory too.

Test 2 – Del­e­gate with very long string

For this test, I’m gonna include a 1000 char­ac­ter long string in the clo­sures being queued so that string objects need to be kept around in mem­ory for as long as the clo­sures are still around. Now let’s see what happens!

   1: // instantiate a basic smt with only one thread in the pool

   2: var threadpool = new SmartThreadPool(new STPStartInfo

   3:                                          {

   4:                                              MaxWorkerThreads = 1,

   5:                                              MinWorkerThreads = 1,

   6:                                          });

   7:  

   8: var queuedItemCount = 0;

   9: try

  10: {

  11:     // keep queuing a new items which just put the one and only thread

  12:     // in the threadpool to sleep for a very long time

  13:     while (true)

  14:     {

  15:         // generate a 1000 character long string, that's 1000 bytes

  16:         var veryLongText = new string(Enumerable.Range(1, 1000).Select(i => 'E').ToArray());

  17:  

  18:         // include the very long string in the closure here

  19:         threadpool.QueueWorkItem(() =>

  20:                                      {

  21:                                          Thread.Sleep(10000000);

  22:                                          Console.WriteLine(veryLongText);

  23:                                      });

  24:         queuedItemCount++;

  25:     }

  26: }

  27: catch (OutOfMemoryException)

  28: {

  29:     Console.WriteLine("OutOfMemoryException caught after queuing {0} work items", queuedItemCount);

  30: }

Unsur­pris­ingly, the mem­ory was ate up even faster this time around and at the end we were only able to queue 782232 work items before we ran out of mem­ory, which is sig­nif­i­cantly lower com­pared to the pre­vi­ous test:

image

Part­ing thoughts…

Besides it being a fun lit­tle exper­i­ment to try out, there is a story here, one that tells of a worst case sce­nario (albeit one that’s highly unlikely but not impos­si­ble) which is worth keep­ing in the back of your mind of when util­is­ing thread pools to deal with highly fre­quent, data intense, block­ing calls.

Share

For those of you who are famil­iar with Reac­tive Exten­sions you should know all about observ­ables already, but did you know that there’s another kind of observ­able sequence – Rx.ConnectableObservable.

The dif­fer­ence between the two types of observ­able sequences is well explained here, in short, a con­nectable observ­able sequence allows you to share the same source sequence of val­ues with mul­ti­ple sub­scribers whilst the nor­mal observ­able sequence gives each sub­scriber its own sequence of val­ues. Whilst in most cases this dif­fer­ence doesn’t have any prac­ti­cal impacts as each sub­scribers are given the same val­ues in the same order, how­ever, con­sider this observ­able sequence of ran­dom num­bers between 0 and 1000:

   1: var maxNumber = 1000;

   2: var observableSource = Rx.Observable.GenerateWithTime(

   3:     Math.random(),                                      // initial state

   4:     function (x) { return true; },                      // condition

   5:     function (x) { return Math.random(); },             // iterator     

   6:     function (x) { return parseInt(x * maxNumber); },   // select

   7:     function (x) { return 1000 });                      // interval

As you can see, each time the iter­a­tor is invoked it’ll gen­er­ate a dif­fer­ent value, hence sub­scribers will receive a dif­fer­ent value each time (see demo below):

   1: // first subscriber

   2: observableSource.Subscribe(function (n) {

   3:     sub1Span.html(n);

   4: });

   5:  

   6: // second subscriber

   7: observableSource.Subscribe(function (n) {

   8:     sub2Span.html(n);

   9: });

Instead, if you want to ensure that all the sub­scribers receive the same val­ues, your best bet is to ‘pub­lish’ the source:

   1: // create a connectable observable from the source

   2: var connectableObservable = observableSource.Publish();

which returns you a con­nectable observ­able that you can then attach sub­scribers to:

   1: // connected subscriber 1

   2: connectableObservable.Subscribe(function (n) {

   3:     connSub1Span.html(n);

   4: });

   5:  

   6: // connected subscriber 2

   7: connectableObservable.Subscribe(function (n) {

   8:     connSub2Span.html(n);

   9: });

and once you ‘con­nect’ to the under­ly­ing source, the sub­scribers will start receiv­ing val­ues from the stream:

   1: connectableObservable.Connect();

Demo

Share

I wrote pre­vi­ously about how you can set up mul­ti­ple observ­able sequences and sub­scribe to them with mul­ti­ple observers and cre­ate a many-to-many rela­tion­ship between them.

Whilst this is a very flex­i­ble model with a clear sep­a­ra­tion of respon­si­bil­i­ties, often it requires more work to set up and is more than what you need for the task at hand. For instance, if you’re receiv­ing a steady stream of inputs and want to log the arrival of new inputs as well as per­form­ing some aggre­ga­tion on them, you don’t nec­es­sar­ily have to cre­ate two sub­scribers for the input but instead make use of the Rx.Observable.Do function.

Like the Rx.Observable.Subscribe func­tion, the Do func­tion can take a sub­scriber, or two up to three func­tion objects to han­dle the onNext, onError and onCom­pleted events, in that order. Unlike the Rx.Observable.Select func­tion, it doesn’t have a return value and there­fore won’t allow you to trans­form the input stream, it’s intended purely for caus­ing side effects.

I’ve put together a quick demo (see below) to illus­trate the use of the Do func­tion in con­junc­tion with other com­mon RxJS func­tions such as Select and Where. For this demo we just need two <span> ele­ments, one to show the run­ning total, the other to show a log mes­sage every time a new value is received:

   1: <body>

   2:     <div id="wrapper">

   3:         <p>Sum of squares of odd numbers received : <span id="sum"></span></p>

   4:         <p><span id="log"></span></p>

   5:     </div>

   6: </body>

And the Javascript to go along with it:

   1: <script type="text/javascript" src="js/jquery/jquery-1.4.4.min.js"></script>

   2: <script type="text/javascript" src="js/rxjs/rx.js"></script>

   3:  

   4: <script type="text/javascript">

   5:     $(function () {

   6:         var logSpan = $("#log"), sumSpan = $("#sum");

   7:  

   8:         // create an observable which gives ten numbers in total at 1 second 

   9:         // interval with a 13% chance of exception being thrown at each interval

  10:         var observable = Rx.Observable.GenerateWithTime(

  11:             1,                                  // initial state

  12:             function (x) { return x <= 10; },   // condition

  13:             function (x) {                      // iterator

  14:                 var prob = Math.random();

  15:                 if (prob < 0.13) {

  16:                     throw "Better luck next time!";

  17:                 }

  18:  

  19:                 return x + 1;

  20:             },

  21:             function (x) { return x; },         // select

  22:             function (x) {                      // interval

  23:                 return x === 0 ? 0 : 1000

  24:             });

  25:  

  26:         var sum = 0;

  27:  

  28:         observable.Do(function (n) {    // onNext

  29:             logSpan.html("Received new input: " + n);

  30:         }, function (err) {             // onError

  31:             logSpan.html("Error: " + err);

  32:         }, function () {               // onCompleted

  33:             logSpan.html("No more inputs");

  34:         }).Where(function (n) {         // filter the input sequence

  35:             return n % 2 != 0;          // odd numbers only

  36:         }).Select(function (n) {        // transform the input sequence

  37:             return n * n;

  38:         }).Subscribe(function (n) {

  39:             sum += n;                   // add the new input to the running total

  40:             sumSpan.html(sum);          // show the new running total

  41:         });

  42:     });

  43: </script>

Cou­ple of things to note here:

  • line 13 – this is a delib­er­ate attempt to give the observ­able sequence a ran­dom chance of except­ing to invoke the onError han­dler spec­i­fied in the Do func­tion on line 30
  • line 28 – the Do func­tion updates the HTML con­tent in the log <span> ele­ment every time it receives a new value
  • line 32 – show a dif­fer­ent log mes­sage when we have exhausted the observ­able sequence
  • line 34 – apply a fil­ter on the observ­able sequence for all sub­se­quent functions
  • line 36 – only odd num­bers are fed to this handler

Demo

If you are lucky enough (or unlucky enough depend­ing on which sce­nario you’re try­ing to test) just refresh the page and try again and hope­fully you have bet­ter luck the sec­ond time around!

Share