Treehouse courses

Learning with Treehouse for only 30 minutes a day can teach you the skills needed to land the job that you’ve been dreaming about.

I can honestly say that besides technical knowledge, Treehouse also gave me a certain mindset, and while I generally am a very ambitious person, with each completed course I felt energized to better myself even more.



Is there any advice you’d like to share with new students who are aspiring developers?

I’ve been only for about 2+ years in this industry and I could already write tens of pages about what should and should not be done by aspiring developers, but here’s a few:

If you’re offered a project you know nothing about, take it, you’ll learn after.
Be passionate about it. If your brain doesn’t get “turned on” by new concepts, libraries, programming languages, you should not be doing it.
Get ready to learn continuously. The web is like the Universe. Ever expanding. Consequently the same has to happen to your knowledge and skill-set.
Teach others and code forward ( code forward is a concept I came up with and it comes from “pay it forward”; it basically means do a few projects for free once in a while for people who deserve it ).
Accept failure as a necessary step in self-betterment.
Research before asking questions and know when to ask. Putting in even hours or days for finding the solution will always be more rewarding in the long-run than asking a question on StackOverflow waiting to be spoon-fed the answer. That being said, asking has its place especially in a team-based environment or if the deadlines are (as they often are) very tight.
Go to as many interviews as you can. If you tell a recruiter (agent) that you’d like to go to the interview even if just for the sake of the experience, they’ll respect you for that, and will do their best to land you an interview. That being said, don’t just rely on recruitment agencies, show some initiative and contact companies on your own too. It might just be the detail that gets you hired.
Finally, keep learning, especially when you feel discouraged.

22 tips for better data science

These tips are provided by Dr Granville, who brings 20 years of varied data-intensive experience working with successful start-ups, small companies across various industries, and eBay, Visa, Microsoft, GE and Wells Fargo.

1.     Leverage external data sources: tweets about your company or your competitors, or data from your vendors (for instance, customizable newsletter eBlast statistics available via vendor dashboards, or via submitting a ticket)

2.     Nuclear physicists, mechanical engineers, and bioinformatics experts can make great data scientists.

3.     State your problem correctly, and use sound metrics to measure yield provided by data science initiatives.

4.     Use the right KPIs (key metrics) and the right data from the beginning, in any project. Changes due to bad foundations are very costly. This requires careful analysis of your data to create useful databases.

5.     Fast delivery is better than extreme accuracy. All data sets are dirty anyway. Find the perfect compromise between perfection and fast return.

6.     With big data, strong signals (extremes) will usually be noise. Here's a solution.

7.     Big data has less value than useful data.

8.     Use big data from third party vendors, for competitive intelligence.

9.     You can build cheap, great, scalable, robust tools pretty fast, without using old-fashioned statistical science. Think about model-free techniques.

10.  Big data is easier and less costly than you think. Get the right tools! Here's how to get started.

11.  Correlation is not causation. This article might help you with this issue.

12.  You don't have to store all your data permanently. Use smart compression techniques, and keep statistical summaries only, for old data. Don't forget to adjust your metrics when your data changes,to keep consistency for trending purposes.

13.  A lot can be done without databases, especially for big data.

14.  Always include EDA and DOE (exploratory analysis / design of experiment) early on in any data science projects. Always create a data dictionary. And follow the traditional life cycle of any data science project.

15.  Data can be used for many purposes:

·        quality assurance

·        to find actionable patterns (stock trading, fraud detection)

·        for resale to your business clients

·        to optimize decisions and processes (operations research)

·        for investigation and discovery (IRS, litigation, fraud detection, root cause analysis)

·        machine-to-machine communication (automated bidding systems, automated driving)

·        predictions (sales forecasts, growth and financial predictions, weather)

16.  Don't dump Excel. Embrace light analytics.

17.  Data + models + gut feelings + intuition is the perfect mix. Don't remove any of these ingredients in your decision process.

18.  Leverage the power of compound metrics: KPIs derived from database fields, that have a far betterpredictive power than the original database metrics. For instance your database might include a single keyword field but does not discriminate between user query and search category (sometimes because data comes from various sources and is blended together). Detect the issue, and create a new metric called keyword type - or data source. Another example is IP address category, a fundamental metric that should be created and added to all digital analytics projects.

19.  When do you need true real time processing? When fraud detection is critical, or when processing sensitive transactional data (credit card fraud detection, 911 calls). Other than that, delayed analytics(with a latency of a few seconds to 24 hours) is good enough.

20.  Make sure your sensitive data is well protected. Make sure your algorithms can not be tampered by criminal hackers or business hackers (spying on your business and stealing everything they can, legally or illegally, and jeopardizing your algorithms - which translates in severe revenue loss). An example of business hacking can be found in section 3 in this article.

21.  Blend multiple models together to detect many types of patterns. Average these models. Here's a simple example of model blending.

22.  Ask the right questions before purchasing software.

Top 10 Big Data Technologies Of Present Times

Over last few years Big Data technologies are getting due attention and there are several trends and innovations in this space in recent times.    Wednesday, October 22, 2014:  Big Data is a concept which is quite broad and comprises several trends and technology developments. Over last few years Big Data technologies are getting due attention and there are several trends and innovations in this space in recent times. Here we'll discuss top ten emerging Big Data technologies. 1. Column-oriented databases:  Traditional databases are excellent in online transaction processing but when it comes to query performance while data volumes grow, these databases fall short on performance. The new column-oriented databases store data and focuses on columns and not rows. It allows huge data compression and faster query times. 2. Streaming Big Data analytics:  There are several projects in this section including Storm, Spark, Data Torrent, Spring XD and SQL Stream. Apache Storm is an open source distributed real-time computation system which simplifies streams of data and real-time processing. Spark is a data processing platform which is compatible with Hadoop. DataTorrent is a real-time streaming platform which enables businesses to perform data processing. Spring XD supports streams for event driven data while SQLStream provides a distributed stream processing platform for streaming analytics, visualization and continuous integration of machine data. 3. Schema-less databases, or NoSQL databases:  This database category includes key-value stores and document stores. This database focuses on storage and retrieval of large volumes of unstructured, semi-structured or even structured data. 4. SQL-in-Hadoop:  This technology includes Apache Hive, Shark, Apache Drill, Presto and Phoenix among many others. It helps in making queries and it also manages large datasets in distributed storage. Shark is a data warehouse system which supports Hive's query language. Apache Drill is an Apache incubation project and it's designed for scalability. It's backed by MapR. Presto is an open source distributed SQL query engine and Phoenix is an open source SQL query engine for Apache Hbase. 5. MapReduce:  It's a programming paradigm which allows massive job execution scalability against thousands of servers or clusters of servers. Its two tasks are Map task and Reduce task. It converts any input dataset into different set of value pairs while reducing set of tuples. 6. Hadoop:  Hadoop is an open source platform for handling Big Data which can work with multiple data sources. It has other applications too and it's largely used for changing data like location-based data from weather or traffic sensors, web-based or social media data or machine-to-machine transactional data. 7. PIG:  PIG brings the Hadoop project close to developers and business users and it's used by Perl like language allowing query execution over data stored on a Hadoop cluster. PIG was a project by Yahoo! But now it's completely open source. 8. Big Data Lambda Architecture:  Lambda Architecture is a hybrid platform which combines real-time data and pre-computed data to provide a near-real time view of the data at all times. Its frameworks include Summingbird by Twitter and Lambdoop. 9. PLATFORA:  It almost copies Hadoop and it requires developer knowledge to operate. It's a platform which turns queries into Hadoop jobs with immediate effect and creates an abstraction layer to simplify the datasets in Hadoop. 10. SkyTree:  It's a high performance machine learning data analytics platform which handles Big Data. It's an essential part of Big Data. Courtesy: TechRepublic and InfoQ  Sanchari Banerjee, EFYTIMES News Network

MH370 flight mystery may have an answer

Data Science perspective: MH370 flight data are crucial

It has been a while and still no answer for the missing airplane of flight MH370. This case may give many possibilities, so guessing by trying each of them is painful and may lead to wrong directions, time consuming and frustrations. So simulating both physical and mathematical model maybe an answer

Mathematical Model

Data Science is newly established profession to solve problems with data insights in high volume of data, so in this case like airplane information systems, weather condition, pattern recognition, path selections etc.

Based in the data above, Data Scientists like me can build specific models to adapt exact or similar scenarios of the missing plane in the line MH370

This may give as good picture what may happen to flight, where may landed, what weather condition we had on that time, what other circumstances occurred and what impact they had over the plane.

Picture: Possible routes

If Malaysian Airport can provide with specific technical information about MH370 airplane, historical data of all MH370 flights, we may find important patterns to solve this mystery. Period between departure and lost signal may give us the distance compared to average distance of all MH370 flights on the same period. Adding weather condition may help us to segments the other MH370 flights that had same weather condition to seek specific scenarios under same weather condition. Route lines are similar to all flights in the same airline, but segmenting them to specific conditions of the missing flight like weather, airplane type, technical conditions, air pressure etc… may lead us to better targets. Technical check before departure may give us information of what have been checked or not and if there is room for possible technical failure. If there is room for failures, we can use data from all technical failure plan crashes to predict the time occurred the failure by that distance also segmented to adapt most the missing flight model.

Big Data technology provides us with the power to analyze big and complicated data sets, and there are plenty of professionals to do so.

Physical model- Simulation

I am not the expert of the field but it can be very smart to build a flight simulation of MH370 based on the mathematical model that we provided here and including other external data that have impact on the fight itself. Sometimes visualization may bring in table other factors that may be decisive in solving the mystery.

These represent alternative approaches to what may help involved institutions to solve this case and I hope they may consider these.

Tableau explain why FC Barcelona is still the best team in Spain (and the whole World)

Introduction

Futbol Club Barcelona, also known as Barça is a professional football club, based in Barcelona, Catalonia, Spain. Founded in 1899 by a group of Swiss, English and Catalan footballers led by Joan Gamper, the club has become a symbol of Catalan culture and Catalanism, hence the motto "Més que un club" (More than a club). Unlike many other football clubs, the supporters own and operate Barcelona. It is the world's second-richest football club in terms of revenue, with an annual turnover of $613 million and the third most valuable sports team, worth $2.6 billion.The official Barcelona anthem is the "Cant del Barça" and it is knows to all 480 000 000 fans around the World.

Team Philosophy

Johan Cruyff and Charly Rexach returned to Barcelona in 1988 and began to install a philosophy that would change the way people play and view football forever. Both Cruyff and Rexach admit that it wasn't a completely new philosophy. Admitting that it was one adapted from ideas given to them by Michels, and one which many believe had been given to Europe by the Hungarian side of the early 1950s. The foundation of this philosophy was, and still is, built upon the basic template of touch, technique, maintaining possession, and stretching the pitch with continuous circulation of the ball (Tiqui-Taka). Elements that, at the time, were not valued by many Barcelona supporters (Hunter, 2012)



The chart down (OptaPro) shows the power of Barcelona over other La Liga teams, its play philosophy of short passes and ball possession.





Team Discipline and attractivness (support by fans)

Learn About Tableau

Why FC Barcelona is better than any team, this season also?!

Learn About Tableau

Conclusion

With this in mind, it seems that a lot of clubs steal ideas from Barcelona that they see on the surface, such as tactics, and implement them in the short term, but fail to intertwine them into their own specifically moulded model. It is one thing to use Barcelona as inspiration, but it must be remembered that Barcelona's philosophy is tailored to THEIR own needs, no one else's, and the coaching and playing staff have grown together surrounded by it. They have lived by it and through it. Due to Barcelona's on field success, many are using various aspects of the Barcelona model to shape their own coaching methods, training programmes, and playing style etc. Often missing the point that a club philosophy needs to be self defined and fully committed to by all. There are many different styles of playing football, Barcelona have their own unique style of playing, but it should be remembered that this playing style is born out of wider and deeper beliefs in the cultural values of their personal, independent philosophy.

Big Data and Data Science Books - A Baker's Dozen

• Posted by Kirk Borne on January 15, 2014 at 12:30pm
• View Blog

Here are 13 informative and inspirational books on Big Data and Data Science.  This is definitely not intended to be a comprehensive list (since a complete list of such readings would itself be a form of "Big Data", and consequently the number of possibilities is a nearly uncountable number!NOTE definition of "uncountable" = an infinite set that contains too many elements to be countable.)

1. Big Data: A Revolution That Will Transform How We Live, Work, and T..., by Viktor Mayer-Schonberger and Kenneth Cukier
2. The Signal and the Noise: Why So Many Predictions Fail-but Some Don't, by Nate Silver
3. Predictive Analytics: The Power to Predict Who Will Click, Buy, Lie..., by Eric Siegel 
4. The Human Face of Big Data, by Rick Smolan and Jennifer Erwitt
5. Data Science for Business: What you need to know about data mining ..., by Foster Provost and Tom Fawcett
6. The Black Swan: The Impact of the Highly Improbable, by Nassim Nicholas Taleb
7. Competing on Analytics: The New Science of Winning, by Thomas H. Davenport and Jeanne G. Harris
8. Super Crunchers: Why Thinking-by-Numbers is the New Way to Be Smart, by Ian Ayres
9. Big Data Marketing: Engage Your Customers More Effectively and Driv..., by Lisa Arthur
10. Journeys to Data Mining: Experiences from 15 Renowned Researchers, by Mohamed Medhat Gaber (editor)
11. The Fourth Paradigm: Data-Intensive Scientific Discovery, by T.Hey, S.Tansley, and K.Tolle (editors)
12. Seven Databases in Seven Weeks: A Guide to Modern Databases and the..., by Eric Redmond and Jim Wilson
13. Data Mining And Predictive Analysis: Intelligence Gathering And Cri..., by Colleen McCue

And here are two more, as a bonus:

 14. A Statistical Guide for the Ethically Perplexed, by Lawrence Hubert and Howard Wainer

 15. Too Big To Ignore: The Business Case for Big Data, by Phil Simon 

Are You Recruiting A Data Scientist, Or Unicorn?

Guest blog by Jeff Bertolucci (InformationWeek)

Many companies need to stop looking for a unicorn and start building a data science team, says CEO of data applications firm Lattice.

The emergence of big data as an insight-generating (and potentially revenue-generating) engine for enterprises has many management teams asking: Do we need an in-house data scientist?

According to Shashi Upadhyay, CEO of Lattice, a big data applications provider, it doesn't make sense for organizations to hire a single data scientist, for a variety of reasons. If your budget can swing it, a data science team is the way to go. If not, data science apps may be the next best thing. "If you look at any industry, the top 10 companies can afford to have data scientists, and they should build data science teams," Upadhyay told InformationWeek in a phone interview.

But the solution is less clear for smaller organizations. "The pattern that I've seen now, having done this for over six years, is that very often medium-sized companies think of the problem as, 'I need to go and get me one data scientist,'" said Upadhyay.

[Guidelines aim to combat potential misuse of big data. Read Data Scientists Create Code Of Professional Conduct.]

But the shortage of data scientists, a problem that's only expected to worsenin the next few years, makes that approach a risky proposition.

For example, a company may hire one or two people, Upadhyay said, "but before you know it, because the supply for this talent group is so far behind demand, they have lost this person [who] has gone to the next company. And all of a sudden, all that good work is lost. And you ask yourself, 'Why did that happen? And how can I manage against it?'"

One common problem, he noted, is that companies simply don't understand data scientists and how they work. The job generally requires knowledge of a wide array of technical disciplines, including analytics, computer science, modeling, and statistics. "They also tend to be fairly conversant in business issues," Upadhyay added.

But it's often difficult to find these divergent skills in a single human being. "It's a little bit like looking for a unicorn," Upadhyay said.

When medium-sized companies -- those that fall below the top five in a given industry, for instance -- hire just one or two data scientists, they often can't provide a long-term career path for those people within the company. As a result, the data scientists get frustrated and move onto the next thing.

In Silicon Valley, where data scientists command six-figure salaries and are in great demand, it's very difficult to retain talented people.

The better solution? Build a team.

"You will absolutely get a benefit if you hire a data science team," said Upadhyay. "Go all the way [and] commit to creating a creating a career path for them. And if you do it that way, you will get the right kind of talent because people will want to work for you."

Smaller companies that can't afford data science teams should consider big data applications instead. The biggest firms -- in Upadhyay's words, "the Dells, HPs, and Microsofts of the world" -- can take both approaches: data science teams and big data apps.

The team approach seems to be winning. "I rarely see teams that are one or two people in size," Upadhyay observed. "Obviously people have those teams, but they tend to evaporate over time. Until they get to a team of 10 people or more, [companies] can't justify it."

So what does a data science team cost, and what's the payoff?

Upadhyay offered this example: Say you hire a team of 10 data scientists with an average annual cost of $150,000 per employee. "That's $1.5 million for a data science team," he said. "So they better be creating at least $15 million dollars in value for you -- 10 times [the expense] -- to be worth it."

Emerging software tools now make analytics feasible -- and cost-effective -- for most companies. Also in the Brave The Big Data Wave issue of InformationWeek: Have doubts about NoSQL consistency? Meet Kyle Kingsbury's Call Me Maybe project. (Free registration required.)

Your Data Analysis Takes How Long?

Reference: www.biblogg.no

Av

Andy Cotgreave

– 13. oktober 2013

Publisert i: Forside, Toppsak, Trender, Utvalgte saker

 

Andy Cotgreave, Social Content Manager at Tableau Software, looks at how analytics tools can help to save valuable time

Imagine, for a moment, that you’ve been given a task to analyse a dataset inside sixty minutes and share your results. How far do you think you would get in that time?
It’s a question I had cause to reflect on recently, after running «Fanalytics», a workshop for users of Tableau Public. In the workshop, we gave people a dataset and one hour to do something cool. Their results were astounding.
To understand why they were so impressive, let me provide a little context by considering how many of us work with data.
First, let me dispel a myth. Contrary to popular opinion, if you are using spreadsheets or traditional BI tools, it is quite possible to build beautiful charts. Unfortunately, each view of your data takes considerable time to build, Do you have that time to spare in your working life? What if the chart you take 10 minutes to build doesn’t answer your question? What if it inspires a new question? You have to go back and start again.
What if you could explore your data at the speed of thought instead? What if each mouse click changed the view instantly? This is what we call visual analytics: it allows you to find insight in your data at speeds unimaginable just a few years ago.
You’re probably wondering how all of this relates to the Fanalytics competition I mentioned earlier. Well, during the session, we gave our teams a list of every UK Number one album since 1956, downloaded from Wikipedia. The instructions they were given were to analyse the data and publish something interesting within one hour.
Did they deliver? Oh, boy, yes, and in ways that made my jaw drop. Each entry was different. The winning team analysed albums that had been to number one more than once, revealing perennially popular music, and the effects of sales on a musician’s death. Another team came up with an album explorer that found out which album was number one on your birthday. One team created a visually gorgeous dashboard, sure to engage anyone. A further team came up with a predictive model based around the likelihood of any album title to get to number one. You can see all the entrants on Tableau’s Fanalytics blog post. What was truly amazing was that they did this in one hour. Sixty minutes!
Unfortunately, many people are stuck with tools that are cumbersome or too hard to use. It often takes more than an hour just to connect to data. The simple lesson I’ve learnt from the recent session is that although some tools can make amazing charts, they are often unnecessarily complicated. With some, you need to fill in five steps in a property wizard just to draw a chart. In others, you are required to write custom scripts before you can start drawing anything.
The question we need to ask is whether we are using the right tools to answer questions quickly and in the most efficient way? If not, then perhaps it’s time to ditch these time hogs and focus on analytic tools that save you time instead!

The Data Science Mindset

Intro 

Names like ‘R’, ‘SQL’, and ‘D3’ make data science seem more like alphabet soup than a deliberate practice of working with data. It’s so easy to get lost in the sea of acronyms, packages, and frameworks that we often find our students prematurely optimizing for the right toolset to use, unable to move forward until they have researched every available option. In reality, data science isn’t just about the tools. It’s a mindset: a way of looking at the world. It’s about taking advantage of our modern computers and all of the information that they’re already collecting to study how things work and push the limits of human knowledge just a little bit further. We have a favorite saying around here — data is everything and everything is data. If we begin with this mindset, a lot of data science approaches naturally follow.

 

Store Everything

Storage is cheap. Collect everything and ask questions later. Store it in the rawest form that is convenient, and don’t worry about how or even when you’re going to analyze it. That part comes later.

Use Existing Data

We’re already storing data — let’s use it. When faced with questions, data scientists regularly adapt the query so that it can be approximately answered with an existing and convenient dataset. The best part of data science is discovering surprising applications of existing stores of data. For example, there is a plethora of satellite imagery of Earth. We can use this data to learn about fertilizer use in Uganda, or use pictures of the Earth at night to estimate rural electrification in developing countries.

Connect Datasets

We’re storing everything, all over the world, inexpensively, for the first time in history. There are many lessons to be learned by utilizing more of this treasure trove. Don’t worry about making the best use out of a single source of data. Focus on connecting disparate datasets rather than tuning your models. Conventional statistics teaches a lot about how to choose analysis methods that are appropriate for your data collection approach and how to tune the models for a specific dataset.
Effective data science is about using a range of datasets, connecting the dots between one set of data and another, such as predicting restaurant health scores based on Yelp reviews. In machine learning speak: it’s often better to collect more features rather than spend days optimizing hyperparameters.

Anything Can Be Quantified

Our culture loves to quantify. If you can turn it into a number, that number can be put into a table. Importantly, that table can now be processed by a computer.
A spreadsheet about sewer overflows is clearly data to most people, but what about a calendar? At first, a calendar might not seem like the sort of data that you analyze with statistics. However, you can also represent a calendar as a spreadsheet and as a graph.

Data science becomes a creative endeavor when peeling away the obvious variables presented to you. Maybe you have a bunch of PDF documents. You could easily extract the text in the PDFs and search through the content. Depending on the problem you are solving, these files hold more interesting information than just the text. You can get the page count, the file size, and the shapes of the pages and the program that created it. There is information hidden in many datasets that goes beyond what’s immediately obvious.
There is a lot of talk about the difference between different kinds of data. There’s “qualitative” vs. “quantitative” and “unstructured” vs. “structured.” To me, there isn’t much difference between “qualitative” and “quantitative” data, nor is there between “unstructured” and “structured” data because I know that I can convert between the different types.
At first, the registration papers of company might not seem like interesting data. They begin as paper, most of the fields are text, and the formats aren’t particularly standardized. But when you put them in a database in a machine-readable format, qualitative data becomes quantitative data that can be used to supplement other data sources.

Send Boring Work to Robots

We no longer live in an era where “computer” refers to someone who carries out calculations. Find yourself doing something over and over? Give it to the bots. As far as data analysis goes, modern computers can be far more effective at rote tasks, such as drawing new graphs with every update of a dataset.
Data collection is a prime example of a task that should be automated. A common scene in university research labs is swaths of grad students handing out paper questionnaires to participants of studies. The data scientist says: collect the data automatically and unobtrusively, using existing systems whenever possible. The supercomputers we carry in our pocket are a great place to start.
This mindset can be applied not only to the data, but also to the process itself. Rather than learning and remembering your entire analysis process, you can write a program that does the whole thing for you, from the original acquisition of the data, to the modeling, to the presentation of results to another person. By making everything a program, you make it easier to find mistakes, to update your analyses, and reproduce your results.

Tools

Once inside the data science mindset, solving interesting problems becomes a function of data acquisition and processing. Computers can fit models and make predictions about datasets that are too big to wrap your head around and convert paper documents into electronic tables. They probably know more about you and your habits than you know yourself! Use the tools available to you, but don’t get caught up on the tools themselves.
Properly discussing these relevant tools is another post (maybe a book), but here’s one thought. While it always helps to have more education, you don’t need a PhD in math or computer science in order to create useful things. Loads of wonderful algorithms have already been implemented for you, and simple algorithms often work quite well. If you’re just getting started, focus on the “plumbing” that connects different datasets and systems together.

Data Science Mindset at Zipfian Academy

Our course teaches many data science tools, but we also teach the data science mindset, because you need both to be a great data scientist. To this end, we organize our 12-week course by projects — such as a recommendation engine or spam filter — rather than software packages or algorithms. We teach the various tools in context of applied projects so students learn how to choose the appropriate tool and how to build the plumbing that connects them.
In the end, it’s not about the newest, trendiest framework or fastest data analysis platform. It’s about finding interesting insights from your data and sharing it with the world. Start small, get your hands dirty, and have fun!