Varshil Gandhi

Placement Experience | Microsoft Data Science

Wed, 24 Jun 2020 00:00:00 +0000

Hi, I am Varshil. I recently graduated from EEE dept, IIT Guwahati and am currently working as a Data & Applied Scientist at Microsoft. While documenting my placement experience, I realized that my story may not fit in the standard experience catalogue of people. I had varying priorities regarding placements & higher education, and the latter usually remained on the top all the time. Nevertheless, this article includes my personal story and lessons I learned throughout the placement season, which would be helpful for students applying for data science profiles in the upcoming placement season.

Why Data Science:

Coming to my experience with the placement process, starting with a clear view of the profiles to apply for is critical. The first question that needs answering is: Why pursue Data Science? And what do these roles fulfil? I personally think anyone having a knack for innovating & engineering products using data through recent advances in machine learning should definitely try. In short, data scientists are the middlemen that bring in state-of-the-art research and engineers it for the product improvisation. Moreover, for students interested in the research aspects of ML, these positions require keeping up with the state-of-the-art techniques and collaborating with research scientists, thus keeping up one’s exposure into the domain & assisting in higher studies.

Pre-Test preparations:

When I started out, I was clear on which companies I would be targeting, and I directed my preparations accordingly. With my decision to go forward with both higher education & placement preparations, I went ahead only with companies having Data Science job profiles & targeted some individual companies working in ML even though the job position doesn’t explicitly mention Data Science. Before the tests, I majorly focused on revising & revisiting machine learning concepts (especially its application aspects), probability & linear algebra, solving puzzles and reviewing ML libraries in Python. Moreover, I also practised some coding problems covering the basics of data structures & algorithms from InterviewBit & GeeksForGeeks. I wasn’t able to devote a lot of time towards coding. Still, it is always better to practice more coding, as there are companies with Data Science profiles that conduct coding tests only. While my preparation in ML was based on 2+ years of experience in the same; it may not be the case for everyone. So, in the points below, I will go over some ‘before test preparation’ tips briefly.

Resume:
- It is a critical part of the placement process. Some go-to tips I followed: Start your resume early on and always try keeping it concise, brief and explanatory; in one-page if possible. Describe your internships and projects such that even a layman can grasp the idea with little understanding of the topic. Avoid using complex terms, code names, etc unless it’s absolutely necessary. If one has multiple projects, only include those on which you can emphasize and show your contribution.
Portfolio:
- For Data Science profiles, projects and internships in ML related domains matter a lot. Having great (personal & internship) projects, open-source contributions in famous ML libraries or packages, excellent Kaggle profile, and research papers will definitely help stand out of the crowd.
Machine Learning:
- One would need a thorough understanding of machine learning algorithms, their theoretical aspects and applications. Along with it, basic knowledge of deep learning would also be a good addition. I used this book as my go-to reference for ML concepts and applications. Apart from this, some companies ask coding problems on simple ML models based on regression and time-series data. For those, one should also require familiarity with machine learning libraries like pandas, scikit-learn, etc.

Test:

For Microsoft’s Data Scientist role, the test was MCQ based, consisting of around 60+ MCQs in 60 minutes. The test covered theoretical & application aspects of several topics in machine learning, such as regression, classification, decision trees, random forests, SVM, neural networks, generative/discriminative models and dimensionality reduction algorithms. Furthermore, it included some questions based on skip connections and language models, thus requiring a brief idea of essential concepts in deep learning. While the Microsoft test was solely MCQ based, some other tests actually needed us to code up solutions for ML problems in a short time frame. I also encountered questions on SQL and R language, but they are occasional and can be anticipated beforehand through the job description. For test preparation, I have shared a brief list of topics at the end of this article that covers up a vast majority of test & interview topics.

Two days before the interviews, around 11 students were shortlisted for the interview process. For the SDE profile, Microsoft has a standard ‘Group Fly’ round a day before the interviews, where they give coding questions to solve and select a handful of students for the interviews. While in my case, we were informed that the ‘group fly’ round was not meant for Data Scientist profiles, and we moved forward to the interview process directly. 12 hours before the interviews, I revisited all my notes on ML algorithms, reviewed my resume and focused primarily on previous interview questions and case studies. Apart from that, I relaxed, chilled out and ate a lot of chocolates. :P

Interview:

Coming to the interviews, the process had 3 rounds, all technical interviews and no HR round. In the first round, the interviewer asked me to choose any machine learning algorithm I liked and inquired more on its theoretical aspects. For some specific questions, he was expecting answers understandable to a layman. Below are some of the questions that were asked:

How would you explain information gain in decision trees?
How would you design an anomaly detection algorithm using variants of decision trees?
Neural Networks & weight matrices, backpropagation
Questions related to Ensemble models and stacking.

The second round was more of a case study. Firstly the interviewer asked me if I have worked on any projects that involve feature engineering. Then, he presented a problem, where one is expected to build a classification system that detects faults or bone cracks in high-resolution X-ray images while optimizing several other factors like time delay, cloud processing, scalability, need for local compute etc. With each idea I propose, he corrected and directed me towards a different aspect. The round went on for about 30 minutes or more on the same question. Though my final answer was not what he had expected, he was happy that I reached pretty close and kept churning out viable solutions.

In the third round, the interviewer focused more on my projects & internships. The interviewer was observant & inquisitive towards my internships. As most of my projects involved reinforcement learning, he asked me about some mathematical formulations regarding the same. Moreover, he was curious about the motivation behind those projects. Finally, he started asking questions on ML algorithms & data pipelines, each more difficult than the last till I wasn’t able to answer. The final question where the interview ended was:

How would you design a clustering algorithm using decision trees?

Overall, all the interviewers were extremely friendly, polite and always entertained questions from my side. What I feel helped me a lot during the interview, apart from the core ML knowledge is my past experience on multiple related projects & internships. Revisiting those learning pathways during interviews adds practicality & strong support for the solution I propose. Apart from all that, keep your interview priorities straight and clear, be attentive & confident, and last but not least, be curious and learn from every experience.

There is no foolproof way to crack data science interviews, but it is with one’s own experience and of others before that can help navigate the unknown. Enjoy the placements, learn from it, help your friends and don’t forget to smile throughout the process. May the force be with you!

List of ML topics to review:

Naive Bayes (Decision boundary)
Linear & Logistic Regression
Multi-class vs Multi-Label Classification
SVM
Decision Trees
Random Forests
Bias Variance Trade-Offs
Time-Series Data Handling
k-Means Clustering
PCA, LDA, t-Sne
L[1-inf] losses and their behaviours
Metrics (Precision, Recall, ROC AUC, F1 Score, PR AUC)
Regularization
Backpropagation
Ensemble Models
Skip connections

Exploring learned Representations for Speech-based Applications

Sun, 21 Jun 2020 03:54:18 +0530

Inverse RL on Mountain Car Slides

Wed, 01 Apr 2020 00:00:00 +0000

IRL Experiment on Mountain Car

Code Application

Overview

Inverse reinforcement learning deals with the case of learning the reward function for a situation or an activity where the optimum behavior is known.

Environment Details

Mountain car env : A car (agent) learning how to get to the top of the cliff in the least timesteps.
The goal of the agent is to reach the goal position , represented by the flag at position = 0.5 units.
The state space is a vector [position, velocity] with
- 1.2 ≤ position ≤ 0.6
- velocity∈[-0.07, 0.07]
- The above implies that the state space is continuous and infinite.

The agent can perform 3 actions 0 ,1 ,2 :
- 0: accelerate in the left
- 1: zero acceleration
- 2: accelerate in the right
Acceleration = 0.001 units
The reward function inbuilt the environment is $R(s,a) = -1$ for each timestep and 0 if it reaches the goal.

Experiment Details

NG- Abbel paper vs Ours (Similarities)

A linear function approximator for reward function with 26 basis functions.
A penalty function with penalty constant = 2 in the updates of linear programming.
Equally spaced Gaussian functions as their reward basis functions.
Reward function depend only on the ‘position’ feature of state.

Differences:

Test in the paper was a naive approach (Algo 2). While ours in Algo 3.
The paper discretized the state space $s= [position,~ velocity]$ into $120*120$ discrete states. This makes the number of states finite.
They created a model based on the discretization of state space. We didn’t.
They evaluated the Linear programming maximization for a bunch of 5000 states.

Results

To be added

Questions?

Ask

Documentation

AnimalAI - Imitation Learning

Wed, 05 Feb 2020 04:11:01 +0530

ARLE- Autonomous Robot for Library Enhancement

Tue, 03 Dec 2019 04:11:01 +0530

BERT on NarrativeQA

Fri, 04 Oct 2019 02:10:38 +0530

Fake News Detector

Fri, 04 Oct 2019 02:03:16 +0530

Policy Gradient Algorithms

Fri, 04 Oct 2019 00:49:00 +0530

Robotic Palm Control

Thu, 03 Oct 2019 03:54:18 +0530

Linear Inverse RL Algorithms

Thu, 03 Oct 2019 03:29:44 +0530

Inverse Reinforcement Learning:

The core concept of reinforcement learning in Markov decision process revolves around the definition of reward values. The agent observes a state, proceed with an action and the environment provides the reward and the next state.

Agent-Environment interaction

However, there are numerous cases where designing reward functions are complicated or ‘Mission Impossible’. For example, we want to train an RL agent to drive a car autonomously. The reward function can be hand-engineered, but creating such functions is often time consuming and sub-optimal. Given such reward functions, the agent may not be able to train for the given task.

In numerous scenarios like the one mentioned above, the human knows the optimal way to perform the task (Expert policy) but do not understand the underlying reward function behind the task. So, we instead think of ways to use expert human policies instead of reward functions for an agent.

The simplest of all is just copying the expert behaviour. Given expert trajectories, we design the learning process from a supervised learning perspective: Learn direct mapping from state to actions. This method is known as Behavior Cloning.
Instead of naive copying, can we recreate the reward function of the environment given expert behaviours? Yes. This approach of imitating an expert through learning the reward functions is known as Inverse Reinforcement Learning.

RL vs Inverse RL

The first IRL algorithm (known as Linear Inverse RL) was published in a paper Algorithms for IRL (Ng & Russel 2000) in which they proposed an iterative algorithm to extract the reward function given optimal/expert behaviour policy for obtaining the goal in that particular environment. We will discuss in short what are the significant problems that Inverse RL algorithms face. But, notations first.

Notations:

Markov decision process : $< S, А, P_{sa}, γ, R >$
- Where $S$ = finite/infinite set of states
- $A$ = action set
- $P_{sa}$ = transition probability matrix
- γ = discount factor
- R = reward function
$π : S➝A $ is th policy function.
$ V^{\pi}(s_t)=E[R(s_t)+R(s_{t+1})] $ Value function
$ Q^π(s, a) = R(s) + γ * E_{s’ \sim P_{sa}}[ V^π(s’) ] $ Action value function
$V^*(s)$ : optimal value function
$Q^*(s, a)$ : optimal state value function

The problem is to find a reward function for an environment given optimal behavior/policy. Now, given certain basic conditions, we can easily prove the theorm mentioned below.

Theorem 1: Let $〈S, А, { Pa }, γ〉$ is known. Then, $π(s)≡a1$ is optimal if and only if for all $a∈ A-a1$ , the reward satisfies the equation $$ (P_{a1} - P_a)(I - γ P_{a1})^{-1} R ≽ 0 $$

See the problem in the above equation?

It’s the degeneracy of reward functions. Each policy has a large set of reward function for which the policy is optimal. The set even includes $R=0 ∀s∈S$. There are even more solutions other than these trivial ones. We do not want these trivial reward functions, as the agent will spend its entire lifetime iterating with 0 reward function and will not learn a thing. So how can we choose reward function from such a vast set of solutions?

Heuristics :

We use some specific heuristics.

The natural way is to choose a reward function such that deviating from the policy on any single step must be as costly as possible. This intuition is represented in mathematical terms, as mentioned below. $$ ∑_{s∈S}( Q^π(s,a_1) - max_{a∈A-a_1}Q^π(s,a) ) $$
If the above condition does give degenerate results, the second condition is to add a penalty coefficient, $-λ || R ||_1$ . It will make sure that the reward function remains zero in most places and non-zero at only some states with high profit.

The net reward search equation becomes: $$ Maximize ∑_i min_{π∈丌} ~ { p( V^{π*}(s’) - V^{π_i}(s’) ) } $$ $$s.t ~~ |α_i| ≤ 1, ~~~~ i =1,2,..d $$ $$ p(x)=x~when~x>0, ~else ~penalty*x $$ which can be solved with linear programming methods if reward functions are assumed to be linear.

The paper explained three possible algorithms varying only w.r.t the underlying conditions.

Expert policy function $\pi(s)$ is known, state space $S$ is finite, and transition probabilities $P_{sa}$ are known. (Too ideal case)
Expert policy function $\pi(s)$ is known, state space $S$ is infinite, and transition probabilities $P_{sa}$ are known. (Too ideal case)
m expert demonstrations/ trajectories are known. Transition probabilities are not known. (The realistic case)

Algorithm flowchart:

LIRL algorithm flowchart

Experiments:

I used the MountainCar-v0 environment of OpenAI gym for training and testing the algorithm.

Expert Agent: Used Q-learning algorithm with linear approximators to train the agent for generating expert policy.(This approach is used in numerous imitation learning problems as it becomes difficult for humans to generate expert policies in simulated environments as Mountain Car.)
Specifics: Detailed specs can be found in the slides.

Refer to the github repository for code and more details.

Learned Reward function

This was solely for educational purposes, so I didn’t try to exactly match the results of the paper. Mentioned in the slides are the differences and similarities of the experiment with the paper. Do let me know by mail or create an issue on github if you find any discrepancy in the code or any other issues.

Huffman Coding in ARM-v8

Wed, 02 Oct 2019 01:05:22 +0530

Device for Ship Detection

Thu, 04 Oct 2018 02:15:11 +0530

An example title

Thu, 28 Jun 2018 00:00:00 +0000

Add your content here…