Solve your genomic data headaches at Bio-IT World

Posted on March 14, 2019
Image of Boston waterfront at night

Bio-IT World Conference & Expo at the Seaport World Trade Center in Boston promises to be the biggest and best yet. Over 3,400 life sciences, clinical, healthcare, and IT professionals from over 40 countries are expected to attend from April 16th to 18th.

Join us on booth #317 to talk about solving the challenges you face with genomic data. We’d be delighted to discuss how the latest developments in our data management technology can help you spend less on genomic data storage. And that means you could spend more on your business goals or research objectives.

If you’re attending Bio-IT World Conference & Expo in Boston, drop by booth #317 or book a meeting in advance of the show to make sure we fit into your schedule. We look forward to seeing you there.

If you’d like to know more about the event or register to attend, visit the website.

If you’d like us to keep you informed with the latest PetaGene news and relevant developments, complete the form below and sign up to our newsletter.

We take your privacy very seriously. We will only ever use the information we collect or receive about you in accordance with our Privacy Policy.

Don’t worry, if you prefer us not to contact you about similar products or services we believe you will be interested in, simply opt out at any time using the unsubscribe buttons in our emails.

Photo by Lance Anderson on Unsplash

PetaGene is hiring

Posted on February 21, 2019

Would you like to join a funded, award-winning and growing Cambridge start-up working in the increasingly vital field of genomic data?   

We are looking for developers and a business support administrator.  For the developer roles you’ll need to be proficient in C/C++ and it would help if you’re comfortable working with algorithms.

For the business support administrator position, we’re looking for someone with experience in a business support role. You will need to be well-organised, persistent and accurate to help make our sales operation as effective as possible.

We’ll be at the Cambridge Network Recruitment Evening on 26th February, so come along to talk about how you could fit into our team.

Full job descriptions are on our careers page, with information on how to apply.  We’d love to hear from you.

 

Photo by Tim Mossholder on Unsplash

HIMSS Annual Conference and Exhibition

Posted on January 18, 2019
Logo for HIMSS conference and exhibition

HIMSS 2019

PetaGene will be attending the HIMSS show in Orlando from 12th to 14th February. It’s a great opportunity to meet and tell us about the challenges you face when storing and working with NGS genomic data.

Come to our presentation titled ‘Scaling Genomics Workloads for Precision Medicine’ on 12th February at 3:30pm. It’s happening on the NetApp booth #2779 .

Come along to learn more about how our dramatic compression ratios combined with the right storage architecture can help you achieve optimum efficiency with genomic data; and enable collaboration through the cloud.

Complete the form on this page to schedule a meeting with us at HIMSS 2019.

To find out more about the event, and register to attend, visit the event’s website.

Why Do Community Driven Genomic Data Standards Matter?

Posted on January 15, 2019
Image of a rowing crew

Genomics is a community driven data science. The ability to exchange data and share results relies on a small number of common file formats; and the software tools to read, process and generate data according to these conventions. Many of the common data formats are represented in flat text files; some pre-date the internet, but they are the glue of genomics because the community supports them as de facto standards.

PetaGene Honors and Respects Community Driven Genomics Standards

PetaGene genomic data compression technology reduces the size of genomic data contained in these common file formats, without any loss of information. The compressed data is stored in a unique binary file format, but we add a transparency technology that enables the compressed data to look and interact as the original data files. Through this innovation we always present PetaGene compressed data in compliance with community accepted standards. Compressed BAM and FASTQ files appear as BAM and FASTQ files; and we provide the option to output CRAM files that are readable by any open-source tool that understands CRAM v3. PetaGene compression will never result in yet-another-file-format.

This design choice was made deliberately. By presenting our hyper-compressed data as native BAM, FASTQ, or CRAM, we honor and respect the efforts of the genomics community to maintain stable data exchange media.

Timeline of developments in genomic data formats

Timeline from 1970 to 2018 showing genomic data formats

A Brief History of A C G T

The Needleman-Wunsch algorithm for global sequence alignment was published in 1970; and the Smith-Waterman algorithm for local alignment was published in 1981. Practical implementations of local alignment and search software were developed in the 1980’s. FASTP was a program for local protein sequence alignment and similarity searches published by Lipman and Pearson in 1985. Soon after, FASTA (1988) and BLAST (1990) were developed to provide fast and sensitive similarity searches for nucleotide and protein sequences. The software could read text files as input that became known as FASTA files. The structure of the text data expected by the FASTA programs became the FASTA format. This format still exists and is the format for the human genome reference sequences.

High throughput sequencing came along with the need to track the confidence of each base call in a re-sequencing result. Each sequencing technology has its own “Quality Score” and kept these scores in a separate file. Eventually sequence and quality scores were merged into one file, the FASTQ (circa 2000). There were competing versions of the format (Sanger, Solexa, and Illumina) until some convergence on quality scores occurred around 2009.

The 1000 Genomes Project drove the development of and consensus around data formats for mapped and aligned reads (SAM) and its binary, compressed form (BAM) by Heng Li and others in 2009. The 1000 Genomes Project also produced the Variant Call Format (VCF) in 2009 which captures variant information for a genome or set of genomes. CRAM is the latest addition as a format for compressed genomic data, introduced in 2011.

These formats became open specifications maintained by the 1000 Genomes Project until the Global Alliance for Genomics and Health (GA4GH) took over stewardship in 2016.

Global Alliance for Genomics & Health logo, the standards setting body for genomic data file formats.

Community Standards


FASTA, FASTQ, BAM and VCF formats have persisted and become de facto standards mainly because they are simple, human readable, and an ecosystem of software tools that process these data has grown over time due to the support of many individuals and groups. There was no process to draft a standard, create an optimal representation, and approve the standard.

The informal process by which these standards have emerged may be their most important values:
  • they are not the product of modern data science but they are universally understood by the community, which is more important;
  • there is a legacy and diversity of tools, algorithms, and pipelines that support these formats; and
  • there is a supportive group of people that will help newcomers and fix bugs when they are identified.

The adoption of new file formats or standards will happen slowly in the genomics user base due to the social and academic dynamics of the community. Due to technical and social inertia, replacing these formats is going to be very difficult for the foreseeable future. Perhaps a more formal standards process, driven by the GA4GH, will provide innovations, but this remains to be demonstrated.

At PetaGene we honor and respect these community standards. PetaGene technology provides extreme compression of genomic data without requiring the adoption of a new data structure or file format. Instead we respect the community standards by providing community compliant interfaces to the compressed data. Introducing yet another file format would provide no benefit to the community and only stifle adoption. This is why our products were engineered to present our compressed data as native FASTQ, BAM. or CRAM files.

How Does PetaGene Genomic Data Compression Support the Standards?

While we compress genomic data beyond what GZIP and CRAM can do, we present the data back as the original BAM or FASTQ files. Users and applications never see the compressed data and never need to interact with the compressed file format. Instead we employ functional interposition with the aid of an LD_PRELOAD library that provides dynamic decompression and format translation for all command line tools, applications, and pipelines. In fact, the filesystem representation of the data is also the original .bam or .fastq file names.

Our compression software also has the option to output CRAM formatted files that are created by our compressor and written to a CRAM 3.0 specification compliant file. As such, any CRAM aware tool or application can read the file without the aid of our decompression library (PetaLink). We are completely interoperable with the community standards and there is no lock-in with our compression technology.

Thus PetaGene compression technology supports the existing standards by providing users with perfectly GA4GH compliant FASTQ, BAM, or CRAM data.

What Are The Benefits of This Approach?

Supporting the community standards makes our technology immediately interoperable with all bioinformatics tools. There are no barriers to adoption and we fit right into the ecosystem of tools and technologies for processing, storing, and retrieving genomic data.

We also eliminate a major integration problem that any new format would create: being compatible with existing tools and avoiding any modification or coding to add the technology into existing workflows. By presenting data back to a tool as data it already knows, we eliminate the integration — it just works straight out of the box. This is essentially zero-code integration.

The last reason is because BAM and FASTQ are stable formats and widely used. There is no need to propose a new format to fit our needs and expect the rest of the community to bend to our will. This does not advance bioinformatics or our business.

Open Access

Our compression technology is not open-source, it is open access. The software requires a commercial license but the basic read-back library (PetaLink) is “open access” such that it is always free and always available via petagene.com. The paid license is required for compression but not decompression. For most applications the files are compressed once to achieve storage savings and then decompressed many times. PetaLink remains free to use after the compression license has expired or depleted.

The cloud edition version of PetaLink has many additional features and requires a license.

Sustainable Commercial Support

PetaGene provides business value by making genomic data smaller and faster. Smaller data files translate into reduced storage costs and more budget for primary research activities. Faster data movement reduces processing time which accelerates discovery or provides a clinical result sooner. In our business model, we earn revenues when clients save time and money.

We charge only for compression and license fees are based on compression savings. Clients recover these fees from only a few months of storage savings. Afterwards, clients accrue 100% savings, month after month. Within one year, clients save an average of 50% in storage costs. In 5 years, these savings are over 10x of our original fee to compress the data.

We use these revenues to provide prompt and responsive support to users, fix defects, and continue to improve the product is a sustainable manner.

PetaGene also provides a fully supported, commercial implementation option for CRAM genomics data compression, should you require CRAM. Our CRAM implementation has some additional features: our reference-free compression, storage of CRAM compliant files, and transparent read-back of CRAM files to BAM with universal support of tools that don’t support CRAM. Our CRAM files can be read and processed by any tools that support CRAM v3. We will support your integration, operations, and provide technical assistance should you ever encounter problems with CRAM.

PetaSuite is a fully supported, commercial option for genomic data compression and a commercially supported implementation of CRAM. We provide clients with full warranty and  support while using our software or using CRAM.

Benefits

  • Extreme compression that saves money.
  • Transparent read-back that eliminates integration and speeds up data transfers.
  • Commercial support for PetaGene and CRAM compression workflows.
  • Auditable and verifiable data integrity for lossless compression.
  • Indemnity Insurance for data loss.

Conclusion

PetaGene technology is 100% compliant with community standards and GA4GH. We can provide you the best compression technology and support even if you choose to store open-source CRAM files.

Start solving your data challenges at ASHG 2018 in San Diego

Posted on September 6, 2018
ASHG 2018 graphic - see us on booth 819
ASHG 2018 Banner - see us on booth 819

Every year the ASHG annual meeting attracts the thought leaders in the field of human genetics. Will you be among them at ASHG 2018 in San Diego? If so, visit us on booth #819 during the three days of the exhibition, October 17th to 19th, to let us know about the data challenges you face.

Our team has extensive experience in health information technologies, systems development and large-scale genomics. We’ll be delighted to discuss how to address the frustrations caused by ever increasing volumes of genomic data.

Those challenges go beyond the simple cost of on-premises or cloud storage. There’s the transfer and analysis times for large data sets to consider too. Speeding those up can give a great boost to your research. Our PetaSuite compression software integrates seamlessly with analysis pipelines, reduces I/O demand and facilitates collaboration.

We’ll also be hosting a session in CoLab Theater 3 on Thursday 18th from 4:00 to 4:15pm to present on how using appropriate compression technology can benefit commercial and research organizations working with genomic data.

If you’re attending ASHG 2018 in San Diego, drop by booth #819 or book a meeting in advance of the show to make sure we fit into your schedule. We look forward to seeing you there.

If you’d like to know more about the ASHG meeting or register to attend, visit the website.

PetaGene at the Front Line Genomics Special Interest Group event

Posted on August 28, 2018

Michael Hultner, SVP Strategy and General Manager, US Operations at PetaGeneMichael Hultner, our SVP Strategy and General Manager, US Operations recently attended a Special Interest Group event (SIG) organised by FrontLine Genomics. SIGs bring together senior-level research, clinical and business professionals from across the genomics community to discuss relevant issues and work towards finding solutions to common problems.

In the session on data security, privacy and consent; the subjects of sharing datasets and security provided opportunities to explain how compression technology can help.  In this blog post Michael shares his insights on how access, safeguarding and cloud storage security relate to compression of genomic data.

Sharing datasets

  • Accessing data is difficult for researchers and can take a long time.

    Lack of easy access to a dataset, or information about it, is a significant reason why research projects are time-consuming. The size of the files is a major factor in this. Genomic files can present challenges which regular data storage systems are not set up to solve. It is possible to store data using compression formats which take into account the specific nature of genomic data. This makes life easier for researchers by decreasing transfer and access times. It is also possible to speed up analysis thanks to lower I/O demands.

Security

  • Data is best protected using standard safeguards.

    While compressing data by itself doesn’t make the files any more or less secure, the benefits of compression can help to enable better security or make adopting best practice simpler. Requiring researchers to travel to where the data is stored in order to access it is a common approach for data stored on-premises.  This means that the organisation holding the data cannot enjoy the benefits of cloud storage. It also places demands on the individual researcher and their institution, whether academic or commercial, that might not be practical.  Compressing genomic data using appropriate tools gives the flexibility to enable data sharing and collaboration without exposing it to avoidable security risks.

 

  • There are still many misconceptions about the security of the cloud.

    Security worries are the reason why some research institutions store their data on their own hard drives. These are then transported to individual laboratories. In the age of GDPR and protected health information, the thought of hard drives containing genomic datasets being transported by individual researchers is probably enough to give data stewards sleepless nights. Despite developments in hard drive technology, it’s an impractical approach for today’s genomic datasets. A better technique would be to use established data storage solutions in the cloud or on-premises. That approach allows appropriate access and sharing protocols to be set up as well as suitable backup and restore options should the worst happen.   In this case, compression reduces the cost of these established storage solutions. And if the right kind of compression is used, there is no need to change existing pipelines or bioinformatics systems.

If you’d like to know more about how PetaGene can help with your genomic data management, use the contact form on the site or contact Michael at michael@petagene.com

Everything you need to know about compression for genomic data

Posted on May 3, 2018
dna-compression

The cost of storing and transferring genomic data is rapidly growing as more and more of it is produced. According to researchers from the University of Illinois at Urbana-Champaign and Cold Spring Harbor Laboratory, genomic data from Next Generation Sequencing (NGS) will outstrip all other forms of big data for the sheer volume of size, yielding even more data than the current champions of big data volume.

Save time and money by compressing genomic data files
  • What options exist to make costs and transfer times more manageable for genomic data files?
  • Does storing data in the cloud present its own specific problems?
  • Should bioinformaticians and IT professionals be looking for benefits beyond merely reducing storage costs and the time taken to transfer genomic data files?
  • Are all compression techniques created equal? Or do compression techniques developed specifically for genomic data yield enhanced benefits?
  • What are the innovations and developments in genomic compression that have made life easier for those working with NGS data?

Read this paper from Frontline Genomics to discover everything you need to know about genomic data compression: http://www.frontlinegenomics.com/magazine/21442/genomic-data-101-2018-edition/

Our Story

Posted on April 5, 2018
pg-blog
PetaGene started as a small hint of an idea: that a team of Cambridge University PhDs could together devise a novel approach to the problem of genomic data storage. Continue reading “Our Story”