The field of big data technology is being watched by more and more companies, and open source has always been the soul of big data technology. With some of the higher expectations and requirements for big data tools in a number of segments, a number of more efficient and targeted big data tools have been born. Here are a few of the most compelling open source big data tools.
First, Hadoop related tools
Hadoop
Apache's Hadoop project has almost equaled big data. It has grown to become a complete ecosystem with many open source tools for highly scalable distributed computing.
Supported operating systems: Windows, Linux, and OS X.
2. Ambari
As part of the Hadoop ecosystem, this Apache project provides an intuitive web-based interface for configuring, managing, and monitoring Hadoop clusters. Some developers want to integrate Ambari's functionality into their own applications, and Ambari also provides them with an API that leverages REST (Representative State Transfer Protocol).
Supported operating systems: Windows, Linux, and OS X.
3. Avro
This Apache project provides a data serialization system with a rich data structure and a compact format. The pattern is defined in JSON, which is easy to integrate with dynamic languages.
Supported operating systems: Independent of the operating system.
4. Cascading
Cascading is a Hadoop-based application development platform. Provide commercial support and training services.
Supported operating systems: Independent of the operating system.
5. Chukwa
Chukwa is based on Hadoop and can collect data from large distributed systems for monitoring. It also contains tools for analyzing and displaying data.
Supported operating systems: Linux and OS X.
6. Flume
Flume can collect log data from other applications and then send that data to Hadoop. The official website claims: "It is powerful, fault-tolerant, and has an optimized reliability mechanism and many failover and recovery mechanisms."
Supported operating systems: Linux and OS X.
7. HBase
HBase is designed for very large tables with billions of rows and millions of columns. It is a distributed database that provides random real-time read/write access to big data. It's a bit like Google's Bigtable, but built on Hadoop and Hadoop Distributed File System (HDFS).
Supported operating systems: Independent of the operating system.
8. Hadoop Distributed File System (HDFS)
HDFS is a file system for Hadoop, but it can also be used as a standalone distributed file system. It is Java based and is fault tolerant, highly scalable and highly configurable.
Supported operating systems: Windows, Linux, and OS X.
9. Hive
Apache Hive is a data warehouse for the Hadoop ecosystem. It allows users to query and manage big data using HiveQL, a SQL-like language.
Supported operating systems: Independent of the operating system.
10. Hivemall
Hivemall combines multiple machine learning algorithms for Hive. It includes a number of highly scalable algorithms for data classification, recursion, recommendation, k nearest neighbors, anomaly detection, and feature hashing.
Supported operating systems: Independent of the operating system.
11. Mahout
According to the official website, the Mahout project aims to "create an environment for rapidly building scalable, high-performance machine learning applications." It includes numerous algorithms for data mining on Hadoop MapReduce, including some for Scala and A novel algorithm for the Spark environment.
Supported operating systems: Independent of the operating system.
12. MapReduce
As an integral part of Hadoop, the MapReduce programming model provides a way to handle large distributed data sets. It was originally developed by Google, but is now also used by several other big data tools introduced in this article, including CouchDB, MongoDB, and Riak.
Supported operating systems: Independent of the operating system.
13. Oozie
This workflow scheduling tool is specifically designed to manage Hadoop tasks. It can trigger tasks by time or according to data availability and integrate with MapReduce, Pig, Hive, Sqoop and many other related tools.
Supported operating systems: Linux and OS X.
14. Pig
Apache Pig is a platform for distributed big data analytics. It relies on a programming language called Pig Latin that has the advantages of simplified parallel programming, optimization, and extensibility.
Supported operating systems: Independent of the operating system.
15. Sqoop
Enterprises often need to transfer data between relational databases and Hadoop, and Sqoop is a tool for accomplishing this task. It can import data into Hive or HBase and export it from Hadoop to a relational database management system (RDBMS).
Supported operating systems: Independent of the operating system.
16. Spark
As an alternative to MapReduce, Spark is a data processing engine. It claims that when used in memory, it is up to 100 times faster than MapReduce; when used on disk, it is up to 10 times faster than MapReduce. It can be used with Hadoop and Apache Mesos, or it can be used independently.
Supported operating systems: Windows, Linux, and OS X.
17. Tez
Built on Apache Hadoop YARN, Tez is "an application framework that allows a complex directed acyclic graph to be built for tasks to process data." It allows Hive and Pig to simplify complex tasks. These tasks originally required multiple steps to complete.
Supported operating systems: Windows, Linux, and OS X.
18. Zookeeper
This big data management tool claims to be "a centralized service that can be used to maintain configuration information, name, provide distributed synchronization, and provide group services." It allows nodes in a Hadoop cluster to coordinate with each other.
Supported operating systems: Linux, Windows (only for development environments) and OS X (only for development environments).
Second, big data analysis platform and tools
19. Disco
Disco was originally developed by Nokia, a distributed computing framework that, like Hadoop, is also based on MapReduce. It includes a distributed file system and a database that supports billions of keys and values.
Supported operating systems: Linux and OS X.
20. HPCC
As an alternative to Hadoop, HPCC is a big data platform that promises to be very fast and scalable. In addition to the free community edition, HPCC Systems offers a fee-based enterprise edition, billing module, training, consulting and other services.
Supported operating systems: Linux.
21. Lumify
Lumify is owned by Altamira Technologies (known as National Security Technology), an open source big data integration, analysis and visualization platform. Just try the demo version at Try.Lumify.io and you can see how it works.
Supported operating systems: Linux.
22. Pandas
The Pandas project includes data structures and data analysis tools based on the Python programming language. It allows organizations to use Python as an alternative to R for big data analytics projects.
Supported operating systems: Windows, Linux, and OS X.
23. Storm
Storm is now an Apache project that provides real-time processing of big data (unlike Hadoop only provides batch task processing). Its users include Twitter, US Weather Channel, WebMD, Alibaba, Yelp, Yahoo Japan, Spotify, Group, Flipboard and many others.
Supported operating systems: Linux.
Third, the database / data warehouse
24. Blazegraph
Blazegraph was formerly known as "Bigdata," a highly scalable, high-performance database. It has both an open source license and a commercial license.
Supported operating systems: Independent of the operating system.
25. Cassandra
Originally developed by Facebook, this NoSQL database has been used by more than 1,500 organizations including Apple, the European Nuclear Research Organization (CERN), Comcast, eBay, GitHub, GoDaddy, Hulu, Instagram, Intuit, Netfilx, Reddit and other institutions. It can support very large-scale clusters; for example, Apple's Cassandra system includes more than 75,000 nodes and has more than 10 PB of data.
Supported operating systems: Independent of the operating system.
26. CouchDB
CouchDB claims to be "a database that fully embraces the Internet", which stores data in JSON documents that can be queried via a web browser and processed in JavaScript. It is easy to use and highly available and scalable on a distributed network.
Supported operating systems: Windows, Linux, OS X, and Android.
27. FlockDB
Developed by Twitter, FlockDB is a very fast and very scalable graphical database that is good at storing social network data. Although it is still available for download, the open source version of this project has not been updated for some time.
Supported operating systems: Independent of the operating system.
28. Hibari
This Erlang-based project claims to be "a distributed, ordered key-value storage system that guarantees strong consistency." Originally developed by Gemini Mobile Technologies, it is now used by several telecom operators in Europe and Asia.
Supported operating systems: Independent of the operating system.
29. Hypertable
Hypertable is a big data database that is compatible with Hadoop and promises high performance. Its users include E-Bay, Baidu, Gao Peng, Yelp and many other Internet companies. Provide commercial support services.
Supported operating systems: Linux and OS X.
30. Impala
Cloudera claims that the SQL-based Impala database is "the leading open source analytics database for Apache Hadoop." It can be downloaded as a standalone product and part of Cloudera's commercial big data product.
Supported operating systems: Linux and OS X.
31. InfoBright Community Edition
Designed for data analysis, InfoBright is a column-oriented database with a high compression ratio. InfoBright.com offers paid products based on the same code and provides support services.
Supported operating systems: Windows and Linux.
32. MongoDB
MongoDB has been downloaded more than 10 million times, which is an extremely popular NoSQL database. Enterprise Edition, support, training and related products and services are available on MongoDB.com.
Supported operating systems: Windows, Linux, OS X, and Solaris.
33. Neo4j
Neo4j claims to be "the fastest and most scalable native graphics database", which promises massive scalability, fast password query performance and improved development efficiency. Users include E-Bay, Pitney Bowes, Wal-Mart, Lufthansa and CrunchBase.
Supported operating systems: Windows and Linux.
34. OrientDB
This multi-model database combines some of the features of the graphical database with some of the features of the document database. Provide services such as fee support, training and consulting.
Supported operating systems: Independent of the operating system.
35. Pivotal Greenplum Database
Pivotal claims that Greenplum is "the best enterprise-class analytics database of its kind" and can perform powerful analysis of large amounts of massive data very quickly. It is part of the Pivotal large database suite.
Supported operating systems: Windows, Linux, and OS X.
36. Riak
Riak is "featured" and comes in two versions: KV is a distributed NoSQL database, and S2 provides object storage for the cloud environment. It has both an open source and a commercial version, as well as support for Spark, Redis and Solr.
Supported operating systems: Linux and OS X.
37. Redis
Redis is now sponsored by Pivotal, a key-value caching and storage system. Provide fee support. Note: Although the project does not officially support Windows, Microsoft has a Windows-derived version on GitHub.
Supported operating systems: Linux.
Fourth, business intelligence
38. Talend Open Studio
Talend has more than 2 million downloads and its open source software provides data integration. The company also develops tools for billing big data, cloud, data integration, application integration and master data management. Its users include American International Group (AIG), Comcast, E-Bay, General Electric, Samsung, Ticketmaster and Verizon.
Supported operating systems: Windows, Linux, and OS X.
39. Jaspersoft
Jaspersoft offers flexible, embeddable business intelligence tools for users including Gaopend, Guanqun Technology, USDA, Ericsson, Time Warner Cable, Olympic Steel, Neslaska University and General Dynamics. In addition to the open source community edition, it also provides a paid report version, Amazon Web Services (AWS), Professional and Enterprise editions.
Supported operating systems: Independent of the operating system.
40. Pentaho
Pentaho is owned by Hitachi Data Systems, Inc., which provides a range of data integration and business analysis tools. Three community editions are available on the official website; visit Pentaho.com for information on paid support.
Supported operating systems: Windows, Linux, and OS X.
41. SpagoBI
Spago is called “open source leader†by market analysts, providing business intelligence, middleware and quality assurance software, as well as a Java EE application development framework. The software is free and open sourced, but it also provides support, consulting, training and other services for a fee.
Supported operating systems: Independent of the operating system.
42. KNIME
KNIME's full name is "Konstanz Information Miner" (Konstanz Information Miner), an open source analysis and reporting platform. Several commercial and open source extensions are provided to enhance its functionality.
Supported operating systems: Windows, Linux, and OS X.
43. BIRT
The full name of BIRT is "business intelligence and reporting tools." It provides a platform for creating visual elements and reports that can be embedded into applications and websites. It is part of the Eclipse community and is supported by Actuate, IBM, and Innovent Solutions.
Supported operating systems: Independent of the operating system.
Fifth, data mining
44.DataMelt
As a follow-up to jHepWork, DataMelt can handle tasks such as mathematical operations, data mining, statistical analysis, and data visualization. It supports Java and related programming languages, including Jython, Groovy, JRuby, and Beanshell.
Supported operating systems: Independent of the operating system.
45. KEEL
KEEL's full name is "knowledge extraction based on evolutionary learning", a Java-based machine learning tool that provides algorithms for a range of big data tasks. It also helps evaluate the effectiveness of algorithms in dealing with recursion, classification, clustering, pattern mining, and similar tasks.
Supported operating systems: Independent of the operating system.
46. ​​Orange
Orange believes that data mining should be "fruitful and fun," whether you have years of experience or just getting into contact with this field. It provides visual programming and Python scripting tools for data visualization and analysis.
Supported operating systems: Windows, Linux, and OS X.
47. RapidMiner
RapidMiner claims to have more than 250,000 users, including PayPal, Deloitte, eBay, Cisco and Volkswagen. It offers a wide range of open source and paid versions, but note that the free open source version only supports data in CSV or Excel format.
Supported operating systems: Independent of the operating system.
48. Rattle
Rattle's full name is "easy to learn and use R analysis tools." It provides a graphical interface to the R programming language that simplifies these processes: building statistical or visual summaries of data, building models, and performing data transformations.
Supported operating systems: Windows, Linux, and OS X.
49. SPMF
SPMF now includes 93 algorithms for sequential pattern mining, association rule mining, item set mining, sequential rule mining, and clustering. It can be used standalone or integrated into other Java-based programs.
Supported operating systems: Independent of the operating system.
50. Weka
The Waikato Knowledge Analysis Environment (Weka) is a set of Java-based machine learning algorithms for data mining. It performs data preprocessing, classification, recursion, clustering, association rules, and visualization.
Supported operating systems: Windows, Linux, and OS X.
Six, the query engine
51. Drill
This Apache project allows users to query for Hadoop, NoSQL databases, and cloud storage services using SQL-based queries. It can be used for data mining and ad hoc queries. It supports a wide range of databases including HBase, MongoDB, MapR-DB, HDFS, MapR-FS, Amazon S3, Azure Blob Storage, Google Cloud Storage and Swift.
Supported operating systems: Windows, Linux, and OS X.
Seven, programming language
52. R
R is similar to the S language and environment and is designed to handle statistical calculations and graphics. It includes an integrated set of big data tools for data processing, calculation, and visualization.
Supported operating systems: Windows, Linux, and OS X.
53. ECL
Enterprise Control Language (ECL) is the language developers use to build big data applications on the HPCC platform. The HPCC Systems official website has an integrated development environment (IDE), tutorials, and many related tools for working with the language.
Supported operating systems: Linux.
Eight, big data search
54. Lucene
Java-based Lucene can perform full-text searches very quickly. According to the official website, it can retrieve more than 150GB of data per hour on modern hardware, and it contains powerful and efficient search algorithms. The development work was sponsored by the Apache Software Foundation.
Supported operating systems: Independent of the operating system.
55. Solr
Based on Apache Lucene, Solr is a highly reliable and highly scalable enterprise search platform. Well-known users include eHarmony, Sears, StubHub, Zappos, Best Buy, AT&T, Instagram, Netflix, Bloomberg and Travelocity.
Supported operating systems: Independent of the operating system.
Nine, in-memory technology
56. Ignite
The Apache project claims to be "a high-performance, integrated, distributed in-memory platform that can be used to perform real-time calculations and processing on large-scale data sets at orders of magnitude faster than traditional disk-based or flash-based technologies. The platform includes data grids, compute grids, service grids, streaming media, Hadoop acceleration, advanced clustering, file systems, messaging, events, and data structures.
Supported operating systems: Independent of the operating system.
57. Terracotta
Terracotta claims that its BigMemory technology is "one of the world's best in-memory data management platforms", claiming to have 2.1 million developers and 250 organizations deploying its software. The company also offers commercial versions of the software, as well as support, consulting and training services.
Supported operating systems: Independent of the operating system.
58. Pivotal GemFire/Geode
Earlier this year, Pivotal announced that it would open source code for key components of its big data suite, including NoSQL databases in GemFire ​​memory. It has submitted a proposal to the Apache Software Foundation to manage the core engine of the GemFire ​​database under the name "Geode". A commercial version of the software is also available.
Supported operating systems: Windows and Linux.
59. GridGain
GridIlin, powered by Apache Ignite, provides an in-memory data structure for fast processing of big data and a Hadoop accelerator based on the same technology. It has both an enterprise version for a fee and a free community version, which includes free basic support.
Supported operating systems: Windows, Linux, and OS X.
60. Infinispan
As a Red Hat JBoss project, Java-based Infinispan is a distributed in-memory data grid. It can be used as a cache, as a high-performance NoSQL database, or as a cluster for many frameworks.
Supported operating systems: Independent of the operating system.
Panasonic Insertion Machine Parts
The Insertion machine can be divided into: a cross-line Insertion machine, an axial component Insertion machine, and a radial component Insertion machine.
Panasonic Insertion Machine Parts include Cutter, Ball Screw, WH Flex Cable, Photo Interrupt, Clinch Lever, Belt, Pallete,Shaft Assy,Guide,Scissors Unit etc.
Panasonic Insertion Machine Parts,Insertion Machine Fiber Sensor,Insertion Machine Solenoid Valve,Insertion Machine Solenoid Valve
Shenzhen Keith Electronic Equipment Co., Ltd. , https://www.aismtks.com