Data promises to be for the 21st century what steam power was for the 18th, electricity for the 19th and hydrocarbons for the 20th; we are truly experiencing a new industrial revolution. We are also experiencing profound changes in the information technology landscape. The data around us has become so vast, so demanding, there is so much and it is so complex, that it is not about the processing speed anymore; we truly need deeper forms of computational intelligence to make sense out of this new universe of information.

In addition to machine learning and data mining, some of the skills that are required of data scientists include statistics, software engineering, linear algebra, programming languages such as Python and Java, and platforms such as Hadoop for advanced analytics.

In response to Joaquin Oliveras

It’s been a huge decade for big data and artificial intelligence (AI), two of the biggest tech trends we’ve seen this century. From data-driven manufacturing to self-driving cars, we’ve witnessed dozens of jaw-dropping, previously unimaginable feats, all thanks to advances in big data analytics and AI.

Not so long ago, businesses across industries often sat on tons of useful, game-changing data, unsure about the many ways they could put it to use to gain competitive advantage. But as methods in machine learning, deep learning, and natural language processing became more advanced while computing power went up, seemingly useless data suddenly began to make sense.

It’s been a huge decade for big data and artificial intelligence (AI), two of the biggest tech trends we’ve seen this century. From data-driven manufacturing to self-driving cars, we’ve witnessed dozens of jaw-dropping, previously unimaginable feats, all thanks to advances in big data analytics and AI.

Because one of the goals of AI developers is to create an artificial system that can fool someone into thinking it is human, the more convincing, personality-consistent generative models have received more attention. These models learn by performing their tasks instead of using a more heuristic approach to match questions to responses (as retrieval based models do), which means they can more easily be misled. Such was the case of Microsoft's Tay. The chatbot turned racist and genocidal after interactions with the public on Twitter helped it learn these traits. Following this, Microsoft took down the bot to make alterations.

Guys a fun fact, I was on a lecture just the other day, there I had the pleasure to listen some info about a company called chaosgroup. They create software for 3D rendeing, in plain words, their programs create serious animations. And in the theme of Big Data I was told that in the latest avengers movie the character called "Thanos" weights thousands and thousands of terabytes of data.

Big data analytics applications enable big data analysts, data scientists, predictive modelers, statisticians and other analytics professionals to analyze growing volumes of structured transaction data, plus other forms of data that are often left untapped by conventional business intelligence and analytics programs. That encompasses a mix of semi-structured and unstructured data -- for example, internet clickstream data, web server logs, social media content, text from customer emails and survey responses, mobile phone records, and machine data captured by sensors connected to the internet of things. 

Big Data technologies are evolving with the exponential rise in data availability. It is time for enterprises to embrace this trend for the better understanding of the customers, better conversions, better decision making, and so much more.  It is important for enterprises to work around these challenges and gain advantages over their competition with more reliable insights.

There is a definite shortage of skilled Big Data professionals available at this time. This has been mentioned by many enterprises seeking to better utilize Big Data and build more effective Data Analysis systems. There is a lack experienced people and certified Data Scientists or Data Analysts available at present, which makes the “number crunching” difficult, and insight building slow.

In response to Alex Tetradze

Careen,

What is RDBMS?  Does it have something to do with databases?

A relational database management system (RDBMS) is a collection of programs and capabilities that enable IT teams and others to create, update, administer and otherwise interact with a relational database. Most commercial RDBMSes use Structured Query Language (SQL) to access the database, although SQL was invented after the initial development of the relational model and is not necessary for its use.

In response to Careen Levi

Big data gets generated in multi terabyte quantities. It changes fast and comes in varieties of forms that are difficult to manage and process using RDBMS or other traditional technologies. Big Data solutions provide the tools, methodologies, and technologies that are used to capture, store, search & analyze the data in seconds to find relationships and insights for innovation and competitive gain that were previously unavailable.

Careen,

What is RDBMS?  Does it have something to do with databases?

Big data gets generated in multi terabyte quantities. It changes fast and comes in varieties of forms that are difficult to manage and process using RDBMS or other traditional technologies. Big Data solutions provide the tools, methodologies, and technologies that are used to capture, store, search & analyze the data in seconds to find relationships and insights for innovation and competitive gain that were previously unavailable.

Data science, data analytics, and machine learning are some of the most in-demand domains in the industry right now. A combination of the right skill sets and real-world experience can help you can secure a strong career in these trending domains.

Here are the essential data science skills:

• Strong knowledge of Python, SAS, R, Scala
• Hands-on experience in SQL database coding
• Ability to work with unstructured data from various sources like video and social media
• Understand multiple analytical functions
• Knowledge of machine learning
   

I want to add a comment on how AI is used in education.  AI can automate grading, giving educators more time. AI can assess students and adapt to their needs, helping them work at their own pace. AI tutors can provide additional support to students, ensuring they stay on track. AI could change where and how students learn, perhaps even replacing some teachers.

Big data can be contrasted with small data, another evolving term that's often used to describe data whose volume and format can be easily used for self-service analytics. A commonly quoted axiom is that "big data is for machines; small data is for people."

In response to Magdalena Novak

Science and research is currently being transformed by the new possibilities big data brings. Take, for example, CERN, the Swiss nuclear physics lab with its Large Hadron Collider, the world’s largest and most powerful particle accelerator. Experiments to unlock the secrets of our universe – how it started and works - generate huge amounts of data. The CERN data center has 65,000 processors to analyze its 30 petabytes of data. However, it uses the computing powers of thousands of computers distributed across 150 data centers worldwide to analyze the data.

This is fascinating! I have always wondered what the purpose of CERN was, I will read more about it.

Industry influencers, academicians, and other prominent stakeholders certainly agree that big data has become a big game changer in most, if not all, types of modern industries over the last few years. As big data continues to permeate our day-to-day lives, there has been a significant shift of focus from the hype surrounding it to finding real value in its use.

In response to Mélina Aubert Porcher

The topic of this discussion is the Road Ahead for big data.  What are some other fields in which big data is slated to make a difference?

Sports.  Most elite sports have now embraced big data analytics. We have the IBM SlamTracker tool for tennis tournaments; we use video analytics that track the performance of every player in a football or baseball game, and sensor technology in sports equipment such as basket balls or golf clubs allows us to get feedback (via smart phones and cloud servers) on our game and how to improve it.

Science and research is currently being transformed by the new possibilities big data brings. Take, for example, CERN, the Swiss nuclear physics lab with its Large Hadron Collider, the world’s largest and most powerful particle accelerator. Experiments to unlock the secrets of our universe – how it started and works - generate huge amounts of data. The CERN data center has 65,000 processors to analyze its 30 petabytes of data. However, it uses the computing powers of thousands of computers distributed across 150 data centers worldwide to analyze the data.

In response to Ramesh Sigrún Rocca

Melina,

Big data is applied heavily in improving security and enabling law enforcement. I am sure you are aware of the revelations that the National Security Agency (NSA) in the U.S. uses big data analytics to foil terrorist plots (and maybe spy on us). Others use big data techniques to detect and prevent cyber attacks. Police forces use big data tools to catch criminals and even predict criminal activity and credit card companies use big data use it to detect fraudulent transactions.

Got one more: optimizing machine performance.  Big data analytics help machines and devices become smarter and more autonomous. For example, big data tools are used to operate Google’s self-driving car. The Toyota Prius is fitted with cameras, GPS as well as powerful computers and sensors to safely drive on the road without the intervention of human beings. Big data tools are also used to optimize energy grids using data from smart meters. We can even use big data tools to optimize the performance of computers and data warehouses.

In response to Mélina Aubert Porcher

The topic of this discussion is the Road Ahead for big data.  What are some other fields in which big data is slated to make a difference?

Melina,

Big data is applied heavily in improving security and enabling law enforcement. I am sure you are aware of the revelations that the National Security Agency (NSA) in the U.S. uses big data analytics to foil terrorist plots (and maybe spy on us). Others use big data techniques to detect and prevent cyber attacks. Police forces use big data tools to catch criminals and even predict criminal activity and credit card companies use big data use it to detect fraudulent transactions.

In response to Mélina Aubert Porcher

The topic of this discussion is the Road Ahead for big data.  What are some other fields in which big data is slated to make a difference?

I thought of one more--city optimization.  Big data is used to improve many aspects of our cities and countries. For example, it allows cities to optimize traffic flows based on real time traffic information as well as social media and weather data. A number of cities are currently piloting big data analytics with the aim of turning themselves into Smart Cities, where the transport infrastructure and utility processes are all joined up. Where a bus would wait for a delayed train and where traffic signals predict traffic volumes and operate to minimize jams.

In response to Mélina Aubert Porcher

The topic of this discussion is the Road Ahead for big data.  What are some other fields in which big data is slated to make a difference?

I think one such area is Financial Trading.  High-Frequency Trading (HFT) is an area where big data finds a lot of use today. Here, big data algorithms are used to make trading decisions. Today, the majority of equity trading now takes place via data algorithms that increasingly take into account signals from social media networks and news websites to make, buy and sell decisions in split seconds.

The topic of this discussion is the Road Ahead for big data.  What are some other fields in which big data is slated to make a difference?

While AI is critical for self-driving cars, the military, commerce, AI-driven SEO and gaming, it’s poised to make the most human impact in medicine and human behavior. Imagine the UN leveraging neural networks and deep learning to discover what helps some communities thrive and others fall behind. Those lessons can then be leveraged into community builders, city planners, grants and projects.

In response to Gaetano Albertini

Big data is used to improve many aspects of our cities and countries. For example, it allows cities to optimise traffic flows based on real time traffic information as well as social media and weather data. A number of cities are currently piloting big data analytics with the aim of turning themselves into Smart Cities, where the transport infrastructure and utility processes are all joined up. Where a bus would wait for a delayed train and where traffic signals predict traffic volumes and operate to minimise jams.

I'd never thought about this but you are right, this is really happening and it's improving our lives. But what happens when technology simplifies our live to a degree when we don't need to go outside, meet with friends and experience nature. Maybe one day we won't even need to walk to get to where we want, not even a simple move and we are there. How is advanced technology going to solve such problems?

Technology such as predictive analytics will never be enough on its own to gain meaningful insights from data.  It is essential to apply specialist industry (and sometimes legal) knowledge to the data analytics to identify the factors which have the most impact on potential outcomes.

In response to Анета Владимирова

@Klas, here is an interesting discussion on some of the challenges of Big Data by Dr. Borne.  They all start with the word V: Volume, Variety, Velocity, Veracity, Validity, Value, Variability, Venue, Vocabulary, and Vagueness :)

Aneta, this was a good post about the challenges of big data.  The definition of big data also involves letters that start with V: You have big data if your data stores have the following characteristics:

• Volume: Big data is any set of data that is so large that the organization that owns it faces challenges related to storing or processing it. In reality, trends like ecommerce, mobility, social media and the Internet of Things (IoT) are generating so much information, that nearly every organization probably meets this criterion.
• Velocity: If your organizations is generating new data at a rapid pace and needs to respond in real time, you have the velocity associated with big data. Most organizations that are involved in ecommerce, social media or IoT satisfy this criterion for big data.
• Variety: If your data resides in many different formats, it has the variety associated with big data. For example, big data stores typically include email messages, word processing documents, images, video and presentations, as well as data that resides in structured relational database management systems (RDBMSes).

Big data is used to improve many aspects of our cities and countries. For example, it allows cities to optimise traffic flows based on real time traffic information as well as social media and weather data. A number of cities are currently piloting big data analytics with the aim of turning themselves into Smart Cities, where the transport infrastructure and utility processes are all joined up. Where a bus would wait for a delayed train and where traffic signals predict traffic volumes and operate to minimise jams.

In response to Professor Dodds

I have heard that Blockchains offer an alternative type of data ownership and data marketplace.  Does anyone know what Blockchains are and how they can be used to build a global data commons?  Thanks.

Hello,

The Convergence Ecosystem will lead to a global data commons. It’s not inevitable, but blockchains and distributed ledgers are disruptive technologies that change the structure of the data value chain. Yes, I know that word is overused, but in this case it is true. The point of value capture in the value chain will change. Instead of web companies capturing value and profit by controlling data, data could be stored on decentralised data file systems and blockchains making it accessible to all, not just a select few platforms that collected it.

Aneta,

Excellent post.  I just want to add one clarification.  The V-based characterizations you have listed represent ten different challenges associated with the main tasks involving big data.  They are: capture, cleaning, curation, integration, storage, processing, indexing, search, sharing, transfer, mining, analysis, and visualization.

I have heard that Blockchains offer an alternative type of data ownership and data marketplace.  Does anyone know what Blockchains are and how they can be used to build a global data commons?  Thanks.

@Klas, here is an interesting discussion on some of the challenges of Big Data by Dr. Borne.  They all start with the word V: Volume, Variety, Velocity, Veracity, Validity, Value, Variability, Venue, Vocabulary, and Vagueness :)

Diversification of opinion is an important issue to Big Data.  We should understand the advantages and challenges involved if we want to preserve and foster democracy in our society.